The goal of this project is to prepare a web crawler for any page you want and map it in python code. There is plenty of solution for crawling between the pages but I wanted prepatr some unique approach of this problem in the easy way without complicated structures and unecessary features. Idea for this project was from one of the interview tasks Global App Testing dev task - senior python developer. I hope you will enjoy this project, feel free to contribute and create issues - all of us want a good quality code and everybody are allowedd to make mistakes.
- Python Versions Tested
cd ~
git clone https://github.com/piobig2871/web_crawler.git
cd ./web_crawler
python3 -m venv venv
. venv/bin/activate
python -m pip install -r requierements.txt
- If you want to check some web page always go to
your_website.com/robots.txtto find out if this is legal