CryptoNews
Dashboard Elements
Project Scripts
CryptoNews is a project based on Flask
and Dash
. The main parts of the project are:
ETL: collecting the latest news about crypto from various news sources to a MySQL
database.
Data Analysis: performing a sentiment analysis on headlines and description of articles, by using vaderSentiment
.
Dashboard: display the average sentiment score for the collected articles and how many were evaluated.
The live dashboard is available on: http://msitapati.pythonanywhere.com
With the use of vaderSentiment
I have analysed headline, description and content extract of all the articles and averaged these scores into one main sentiment score for each hour. This score goes from -100% for the most negative possible news sentiment up to +100% for a perfectly positive sentiment score on the news.
To improve the readability of the timeseries I also included a 12h moving average linechart with fill to zero.
At the bottom of the graph there is a simple barchart displaying the count for how many times a crypto-related keyword has been mentioned in news headlines for a given day.
This is a simple data table that displays the news articles metadata and the sentiment score for each article to examine.
As the time series, this table also gets updated when the user selects a certain keyowrd or date range at the top of the dashboard.
This is how the data moves within this project, so that you can see the final resulting dashboard.
main.py
: main script that runs the ETL and Data Analysis scripts every 3600 seconds (1 hour)extract.py
: fetching the latest news from newsapi.org for the keywords: crypto, bitcoin and ethereumtransform.py
: data parsingload.py
: load parsed data into MySQL server
db_dump.py
: convert a complete copy of the artcles table into a DataFrame and dumps it into .pkl formatdata_analysis.py
: performs: data cleaning, Sentiment Analysis and prepares a daily resampling of the data
dash_app.py
: script to setup the layout and plotly graphs necessary to generate the dashboardprototypes
folder: contains scripts of past dashboard versions and future versions that I'm currently working on