A platform to analyze technologies trend over time, by country, and by importance on StackOverFlow.
Fig. 1: Jobservatory Architecture
Under the scraper folder, there is all the code related to data pipelines which have the task of ingesting data into our database. Currently we are using MongoDB as our DB engine, because schema evolution flexibility and the fast iterations that are possible under this model. We are currently scraping data from two sources: Linkedin and StackOverflow.
Our backend is an express server (Nodejs), from which we implement endpoints for several queries. The related code is under jobservatory_express directory. We also cache requests under the assumption that data will not vary faster than 10 minutes, so we have a redis cache to improve latency. Our performance tests showed that the reduction in latency was from 400 ms to 3 ms on average.
As the name suggests, our frontend is built on React framework. Code is under jobservatory_react. For manage the states inside the react app we use Redux, for UI layouts in general Chakra-ui, and for charts we use Nivo. Tests are under react-testing-library reaching 70%.
