woodyk / creeper

creeper

---

Creeper is a simple web crawler.

Usage

./creeper.py  -u [seed url] [options]
	-u	Seed URL to start with.
	-h	This help output.
	-e	Elastic search host.
	-p	Sticky to URL path.
	-d	Enable dynamic page processing.
	-f	Make creeper follow outside links.
	-c	Clear out visited and unvisited domains lists.