The entire application is governed through the docker-compose.yml file and is built with docker compose:
- STEP 1: Install
dockeranddocker-compose, this may need a restart of your system since Docker is a very low level program. - STEP 2: Run
docker-compose up --buildas either a user with permissions to docker, or withsudo/doas, the build flag is required if the backend or frontend code has been changed, additionally-dwill make it detach from the terminal. - STEP 3: ???
- STEP 4: PROFIT!!!
Docker may appear to hang on the
scrapercontainer, this is to be expected since it is downloading 5 courses a second of the ~3.5k and it may take up to 15 minutes to finish scraping since we want to be nice to KU's surprisingly fragile IT infrastructure.
Our backend is built with Clojure, a functional programmering language based on Lisp which runs on the Java Virtual Machine.
This part of the project is responsible for gluing our web-scraper together with the frontend and the database. Furthermore, it is responsible for constructing our SQL queries that are served via the API.
The "scraper" confusingly runs the parser, It was previously responsible for scraping all of the HTMLs and caching them for future use, the parser will convert all the HTMLs into JSON so they can be inserted into the database. The real scraper is now integrated into db-manager
TODO: This entire module is strongly overdue a complete rework into a lighter language (Rust, Haskell, Clojure) since the module is bigger than the entire project combined when built, and takes a long time to parse HTMLs.
Frontend is built in Svelte/Typescript.
Another module written in Clojure, this module is currently not automatically started by the docker-compose and should be run manually. It is responsible for scraping STADS for the statistics on all courses to add them to the returned JSONs from db-manager.