web-crawler-analysis

First Start

install maven
install mysql database (on localhost!)
you may need to install the Spring Framework
setup configfile: ./Analysis/src/main/resources/database.properties
setup configfile: ./HttpCrawler4j/src/database.properties
config seeds: ./HttpCrawler4j/src/seeds.txt
config whitelist (urls that start with these are always crawled): ./HttpCrawler4j/src/url-whitelist.txt
Verify that on building projects, all these config files must be deployed
Include all other config files, e.g. in ./crawler4j/src/main/resources/
On the first run of the HttpCrawler4j the database is generated automatically
In the file ./sql-commands.txt you find usefull SQL commands.

crawler4j: fork of the crawler4j project by Yasser Ganjisaffar with minor changes
HttpCrawler4jDatabase: database configuration using Resthub
HttpCrawler4j: Executable project that crawls the web and stores Metadata to database
Analysis: After Crawling you can use this project to analyse the crawled data. The results are exported to html files.

For Licence details please see licence.txt in the root folder and in each subfolder.

Check database.properties in both folders: HttpCrawler4j and analysis!
Check if MySQL is running: mysql -u crawler -o crawler
mvn clean install (on crawler4j and HttpCrawlerDatabase)
mvn eclipse:eclipse (on analysis and HttpCralwer4j)
if you do not use eclipse: "mvn install" in each folder may be enough
make sure the config files are added to the build (Eclipse: right click on project -> Properties -> Java Build Path -> Source check included and excluded for the files where the config files are!) Else you could get internal errors in the project or errors with RestHub)
analysis-program needs write permission to a given folder (e.g. /analysis/export/)
other maven problems: Maven Problems solved: do exactly what they do there: http://www.avajava.com/tutorials/lessons/how-do-i-add-a-project-as-a-dependency-of-another-project.html?page=2
Hint: do not use brackets () in your (absolute) path or maven builds will fail.

Every Host is only once crawled as the link is found in any other page but only if it's the first visit to this host.
Except if the URL starts with a String in the url-whitelist.txt (not case sensititve)
If you get a http URL to the links, we also try to visit the page as https.
The crawler starts with the seeds given in seeds.txt
You may want to have this seeds in the whitelist.
We respect the robots.txt file. If this file forbids our crawler, we do not use it.
We have a politeness delay (niceWaitTime): the host is not visited twice until this time interval (set in HttpAnalysisCrawler - the other one is not working!).
The crawler follows redirects.
It is possible to set a max depth of crawling (deactivated). (set in HttpAnalysisCrawlController)
The socket timeout and the connection timeout can be set in HttpAnalysisCrawlController.
Crawler is multithreaded. Has still to be tested how many pages at once are feasible.
different priorities for HTTPS/HTTP-pages are possible and respected. Also a different priority for the Whitelist. I used 10 for the whitelist and 5 for http/https. Hence the whitelisted-urls are visited last and the queue wont get too large.