The web scraping process will begin in the "initurl" . It will retrieve all URLs from the web, filter out any unwanted URLs, and store them in a database. Then, a batch of random URLs will be selected from the database for further scraping. To handle this process efficiently, multiple threads will be utilized.
There are two classes involved in this process: "Controller" and "Demo". The "Controller" class is responsible for managing the threading operations and db manage, while the "Demo" class handles the extraction of text and the filtering of links from the scraped data which is ina function called by "Controller"
This project is a demo of how to use a Python script to do webscrap with fault tolerance. Current there is no package provide, user can start from demo.py
- Clone the repository to your local machine.
- Create a virtual environment using
python -m venv venv. - Activate the virtual environment using
source venv/bin/activate - Install the required packages using
pip install -r requirements.txt. - Create a dir for storage all data you wish to extracted. One dir per one websiter per one .py script
- download firefox and web driver (other webdriver also should be work)
- Modify the path in
scriptto point to the correct file location of .db .log - Run the script using
python demo.pywith or without args - The output will be saved in dir and log and db
- Modify the TODO list as needed in demo.py (refering to the url, dir .etc)
- Save the file and run
python demo.py arg1 arg2 arg3to see the updated list.
arg1 : number of round
arg2 : page per round
arg3 : time interval
Make use of .sh in script directory to run job in background and in batch by nohup