oxylabs / pricing-data-collection-from-ecommerce-stores

Appache Airflow DAGs for e-commerce pricing collection.

Pricing Data Collection from E-Commerce Stores

During the webinar on “Driving E-Commerce Success Through Pricing Data: Are You Getting It Right?”, our Engineering Manager, Povilas Kudriavcevas, showcased how to collect product pricing data from https://books.toscrape.com, using Oxylabs Web Crawler and Web Scraper API. In this repository, you will find the Apache Airflow DAGs he used during the webinar.

Apache Airflow setup

Follow official documentation https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html

Setup

1. Copy the contents of the files into your Apache Airflow project.
2. Configure your Oxylabs credentials in the settings.py file.
3. Specify the absolute path to the database.db file located in the dags folder.
4. Execute the DAGs using the Apache Airflow UI.

Watch the webinar to learn more about pricing data collection and follow the steps Povilas took to do it successfully.

The webinar recording will be uploaded soon on our web, next to all other web scraping-related webinars. If you stumbled here before the 10th of May, you can still watch this webinar live. Make sure to register!

We wish you a smooth data collection, and if you have any questions, feel free to reach out to us at [e-mail] or by live chat on our dashboard.