Integration of Django with a python web-crawler. The "CrawlerLogic" file was not written by me. I used it with the permission of the owner.
Site that uses crawling to generate graph of a site. Few examples:
https://gyazo.com/282930148a136646aec0527b367f180b
https://gyazo.com/b8edd9c9183cb4fc90ececcabe2d5a1f
https://gyazo.com/6f80a29febe90169400aa2ad1f7a495e
The graphs are generated with cytoscape.js. Url field is with validation. There are two options to crawl: shallow and deep. Depending on which one you choose, different number of pages will show on the graph.
Shallow crawl: https://gyazo.com/089cb816dd03478a696cbba1dc86182a
Deep crawl: https://gyazo.com/42c627d041d141b33c06dfb4331636ae