Web Crawler to crawl list of given urls and fetch the content of the page
./fetch https://www.google.com https://www.atlassian.com/software/jira
Make sure to bundle install before running tests
rspec web_crawler_spec.rb
- Fetches and saves web pages from provided URLs
- Metadata collection (enabled by default)
- Normalize URLs to avoid crawling the same page multiple times i.e. https://www.google.com and www.google.com are same pages
- Respect robots.txt files
- Improve file naming to avoid collisions
Metadata collection is enabled by default without an explicit --metadata flag.