Web Flayer
Web Flayer is a data mining tool that makes use of the Salt loader system to determine which sites to process, and how. Think of it as wget with plugins.
Requirements
The following python libraries are required:
- beautifulsoup4
- psycopg2
- pyyaml
- requests
- salt
- termcolor
Web Flayer also requires PostgreSQL. Schema can be found in the schema directory.