0MeMo07 / Web-Crawler

Web Crawler with Python

Python Web Crawler

This Python project can be used as a corresponding web crawler from a specific URL. This browser monitors its features while navigating on the given URL and keeps detailed logs for each URL visited.

Features

• Searches a large number of HTML from a URL.
• Finds links in the explored HTML content and adds URLs to visit them.
• You can set the maximum depth level.
• Keeps a list of visited URLs and does not revisit the same URL.
• There are appropriate error message and exception handling elements for error handling.
• Uses color logging.

Use

1. Clone the project:

   git clone https://github.com/0MeMo07/Web-Crawler.git

2. Go to the project directory:

   cd Web-Crawler

3. Install required dependencies:

   pip install -r requirements.txt

4. Run the crawler Python file::

   python crawler.py
   

Support me