mukeshkdangi / crawler_nypost

Crawling web pages and indexing for solr search

nypost_crawler

In this assignment, we worked with a simple web crawler to measure aspects of a crawl, study the characteristics of the crawl, download web pages from the crawl and gather webpage metadata, all from pre-selected news websites.