Problem statement: Language: Python Official site: http://python.org/ Beginer's guide: http://wiki.python.org/moin/BeginnersGuide Tutorial: http://docs.python.org/tutorial/ App Category: Networking DB: None Simple part: Create a web crawler app in python which, given a url seed, can crawl through all links on the page and scan deep for a given level of depth. While crawling the app should be able to return the url page containing a specific search text. Input: 1 - Url seed e.g. www.hackernews.com 2 - Depth e.g. 5 (this means go into links on a page till 5 levels) 3 - search text e.g. "python" Output: the list of url that contains the specified text The Simple part is mandatory to be completed. If you finish the simple part and are eager to take up something challenging, then here's a little complex angle to the problem: Complex part: Write rules around the app for searching. Rule 1: The return Url should contain a specific substring Rule 2: Highlight in output if the url is amongst a long list of blacklisted urls (about 10000 blacklisted urls) Rule 3: Search for multiple search strings and rank Urls as per the number of different search strings found and occurances of each search string in the page Rule 4: Rank as per level of the Url w.r.t. seed url