This project aimed to create a small-scale web app that uses a model developed in Python for data processing and displays its results in an app built using modern web-development tools like: NodeJS, React and D3js.
During my two years working as a data scientist in London, I visited various professional Meet-Ups, where I came aross two communities: data scientists and web developers. It was rather striking how these two groups keep separate. Overall, I think this is due to the fact that the technologies used by each group do not overlap.
The data science community, on the most basic level, often develops its models in Python using highly popular libraries like scikit-learn, pandas, NTLK, etc.
At the same time, web developers' most popular programming language is Javascript and their stack includes NodeJS, Express, React, Angular, etc.
I therefore wanted to develop a project that bridges the divide between data science and web development.
Web technology is therefore used to make data analysis available to the general public.
The search bar on the screen allows you to search the Guardian articles (using Guardian Api). The top-10 most relevant articles are displayed on the left hand side. By pressing the "press me" button, the backend receives the article id and requests its text, processes it in Python and sends the results back to the frontend, which are displayed in the form of bubbles on the screen (using D3js library).
- Express
- React
- D3js
- NodeJS
- JupyterNotebook
Currently, the project is not deployed online. However, it could be run locally.
To start the project you will need to register with the Guardian to receive the dev key
You will have to add this key in two places:
- Add config.env file to
project_PyNode
| client
| ...
| config.env
inside that file: GUARDIAN_KEY=insert_dev_key_here
- Add token.js file inside project_PyNode -> client -> src folder:
project_PyNode
| client
| ...
└───── src
| | components
| | ...
| | token.js
with the following
export const GUARDIAN_KEY = "insert_dev_key here"
You will need to run npm install
twice as well
$...project_PyNode/npm install
and
$...project_PyNode/client/npm install
The application uses Python3 and the following libraries should be installed (use pip
or conda
):
numpy
andnltk
Finally, in the terminal run the following command:
project_PyNode$npm run dev
- Stage 1 The Python code will be invoked as a child process on the backend
- cd ./client then "npm run build", then return to the root "cd ..".
- comment out "/build" in the gitigonre in the root.
- comment out "/build" and "token" in the gitigonre in the root of the client.