connellboyce / simple-web-scraper

App to scrape data from websites

Basic Spring Boot Scraper

App to scrape pepper information from ChilePlanet

Table of Contents

1. Installation and Usage
2. Dependencies
3. Notes

Installation and Usage

Clone the repository

$ git clone https://github.com/connellboyce/spring-boot-scraper.git

Change directory to spring-boot-scraper

$ cd spring-boot-scraper

Clean and build the project with Maven

$ mvn clean

$ mvn install

Run the project through Maven and Spring Boot

$ mvn spring-boot:run

Dependencies

• Spring Boot Starter
• Spring Boot Devtools
• Spring Boot Starter Test
• JUnit Vintage
• JSoup
• Apache Commons Lang3
• Jackson Mapper ASL

Notes

• Because we had to process ~170 inputs, I wanted to expedite the process by using multiple threads with an ExecutorService.
  • Used 25 threads and cut the time by almost 4 times