Terraminator / Index

Create an index of the web

Index

 __  .__   __.  _______   __________   ___ 
|  | |  \ |  | |       \ |   ____\  \ /  / 
|  | |   \|  | |  .--.  ||  |__   \  V  /  
|  | |  . `  | |  |  |  ||   __|   >   <   
|  | |  |\   | |  '--'  ||  |____ /  .  \  
|__| |__| \__| |_______/ |_______/__/ \__\ 
                                                                                                                                                     

Create an index of the web with python

Setup

You just have to install a Python version >= 3 and change self.urls to your start urls for example:

 self.urls = {"www.wikipedia.org", } 

then just run

 python3 main.py 

this will create a sqlite3 db, which contains the urls that has been crawled. To get your index you can run

 python read_index.py 

where you can enter your sql commands.

Ascii Art

The Ascii Art is made with: https://patorjk.com/software/taag/

Disclaimer

This software is only intended for learning purposes and I do not take any responsibility for misuse.

This repository is for academic purposes, the use of this software is your responsibility.