This project can be used as the base project to be used with our Python Scrapy Playbook/YouTube, Javascript tutorials
Scrapy Playwright Guide: Render & Scrape JS Heavy Websites In this guide we show you how to use Scrapy Playwright to render and scrape Javascript heavy websites.
Scrapy Splash Guide: A JS Rendering Service For Web Scraping In this guide we show you how to setup and use Scrapy Splash in your Spider to extract JS rendered data from webpages.
Scrapy Selenium Guide: Integrating Selenium Into Your Scrapy Spiders In this guide we show you how to setup and use Scrapy Selenium in your Spider to extract JS rendered data from webpages.
In most of the videos we use the commands below to
- Clone this project:
git clone https://github.com/python-scrapy-playbook/quotes-js-project.git - Create a Python Virtual Environment:
python3 -m venv venv - Activate the Python Virtual Environment:
source venv/bin/activate - Install Scrapy using pip:
pip install scrapy - Listing the scrapy projects
scrapy list - Running the scrapy project:
scrapy crawl quotes