mkshkmrgpt / WebCrawler

A basic crawler to get site map of website for give depth using Kotlin and Springboot

Web crawler

Basic web crawler Spring Boot app.

Requirements

For building and running the application you need:

• JDK 1.8
• Kotlin 1.2.70
• Maven 3

Running the application locally

There are several ways to run a Spring Boot application on your local machine. One way is to execute the main method in the com.mukesh.webcrawler class from your IDE.

Alternatively you can use the Spring Boot Maven plugin like so:

mvn spring-boot:run

To crawl your website use this location

http://localhost:8080/crawl?site=<SITE_NAME>

Trade offs

• List all the links on given site and does not visit sub-domain links

• List all only the Images media types

• Does not lists any imports links

TODO Improvement

• Links available on site could be arranged more elegantly by following site depth
• Asynchronous calls could be used to visit all inner links