a simple python web crawler.
##How to use Please make sure to install requirements before using the program:
pip install -r requirements.txt
##To do
- Get links from html to crawl them recursively
- Adding Algerian universities official websites to data/algerian_univ.txt
- Read websites from DB