homelinen / ruby-webcrawler

A simple web crawler that outputs a set of linked HTML files representing a site map.

homelinen/ruby-webcrawler Issues

Use Selenium to load Links Added by Javascript
Updated 10 years ago
Webpages should have an Identifier if they are a clone?
Closed 10 years ago1
Consider Pages With Alternate names
Updated 10 years ago
Add http/https if not given at command line
Updated 10 years ago
Change == To not count differing children as the same
Updated 10 years ago
Print out the tree in some maneageable form
Closed 10 years ago1
Add Assets to the Tree
Closed 10 years ago
Deal with 404s and Other Errors
Updated 10 years ago
Implement the Tree Like Structure
Closed 10 years ago
Add a Proper Bin
Closed 10 years ago
Multithread the website grabbing
Updated 10 years ago