This project is a basic one that you can use to web scrape and store the data in your postgres db. The spiders are already setup for a basic tutorial, your job is to fill in the postgres details in the pipelines.py file.
I use playwright to render js.
As you
- Create a virutal environment
python -m venv .venv
- Install necessary packages
pip install -r requirements.txt
- cd into tutorial folder
cd tutorial
- make changes to pipelines.py file and enter necessary db details. Customize the overall project as you may see fit.