Basic Idea of this python bot is to extract all URLs present on a website.
Clone this Repo
git clone https://github.com/hassanaftab93/Web-Crawler.git
Create a Virtual Environment for this Project
python -m venv venv
Activate Virtual Environment for this Project
Windows:
source venv/Scripts/activate
Linux:
source ./venv/bin/activate
Install the Required Libraries
pip install -r requirements.txt
Run the File
sh crawl.sh
Contributions are always welcome!