Lioo7 / Crawler

Multi-threaded Python website crawler that generates a file with all the domestic links on a given website.

Crawler

Multi-threaded Python website crawler that generates a file with all the domestic links on a given website.

How To Run:

1. Open the file main.py and execute the main function.
2. Enter the name of the project.
3. Enter the URL that you would like to crawl.
4. Wait for the spiders to finish crawling.
5. You can view all URLs in the crawled.txt file.