cit-e "site-eee" is a site crawler that I've written to analyze websites for HackerOne.
This crawler can be polite and listen to the robot.txt, or it can be rude and scan everything.
Coming features:
- get all input on every page
- get all output on every page
- outputs are values that will differ between users
- if you would like other features, please post it as a feature-request under issues.