domeniqque-pereira-deel / mk-webscraper

Data extraction tool to get products information from Mary Kay's website. Download images and meta data. Automatic upload your assets to SWS S3 if you want.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

MK Web Scraper

A tool to extract products informations of the MK public site.

It will download all the products and meta data like, price, description, colors, and product types in a well structured and versioned .json file (example).

If you want, you'll be able to automatic upload each image and the final .json file to your AWS S3.

I cannot be held responsible for how you will use this tool. Please read the LICENSE file.

This tool was used for study purposes only.

Instalation

First install the dependencies using yarn install. So, in the src/config.ts file, uncomment the product types you want to get the data.

Finally, run yarn start:dev to start the process.

The .json file with all the products will be saved in the output directory.

Automatic Upload

If you want to upload the data to your AWS S3 bucket, you'll need to set your credentials in the .env file and change the IMG_UPLOAD_TO_S3 variable to true.

cp .env.example .env

Before the upload, all images will be optimized using the sharp package.

About

Data extraction tool to get products information from Mary Kay's website. Download images and meta data. Automatic upload your assets to SWS S3 if you want.

License:MIT License


Languages

Language:TypeScript 99.3%Language:Shell 0.7%