In this project, An API scraper is implemented by using Udemy Restful APIs that are providing course and instructor data in a specific category, subcategory and topic. The implemented API scraper store the course and instructor data in MongoDB. The API Scraper automatically starts to scrape the data after SpringBoot get started. It provides the error handling while requesting data from Udemy Restful APIs. It uses a simple queue that can transition between 3 states: Queued, Complete, Failed. Each api request is queued and if any of them is failed, It will request them later. The requesting data is quite large so that It makes multiple API requests happen at the same time by using multiple threads. Before each request is done, the thread is slept because of Rate Limiting issues. It also logs each request done using Log4j.
In the project RiveScript is used as a chatbot library. Lombok framework is also used for simple looking classes by not writing getter, setter, equals methods. As testing frameworks, JUnit 5 and Mockito are used.
Parameters of the Aplication can be easily changed with changing application.properties file. Each parameter is explained below:
server.port: application port
api.request.proxy.url: If there is a proxy in the network, It is supposed to be set the proxy url for requesting Udemy Restful APIs
api.request.proxy.port: If there is a proxy in the network, It is supposed to be set the proxy port for requesting Udemy Restful APIs
api.request.thread.number: It sets how many multiple API requests can be made happen at the same time
api.request.thread.sleep.time: It sets the duration sleep of threads
scraper.page.size: It sets how many courses can be got in one request
scraper.limited.data: It is used for testing smaller of data
chatbot.suggested.course.number: It sets how many suggested course will be returned to the user
After setting the parameters of the aplication, The application can be easily up running the boot class named ChatbotApplication. The application is also dockerized.
Therefore, the application docker image can be created using Maven command that is mvn package docker:build.
There is a docker-compose.yml in src/main/docker. The application can be easily up running the docker command that is docker-compose up -d in src/main/docker
The application can answer the questions. For instance the questions can be like these:
/chatbot/List web development courses
/chatbot/Show me free lectures
/chatbot/I want to learn finance
/chatbot/I am looking for PHP courses
/chatbot/Would you list Finance classes?
/chatbot/Show me the best seller java courses