- Introduction
- Installation Guide
- Dataset
- File Preparation
- Data Retrieval
- Data Preprocessing
- Data Labeling
- Feature Extraction
- Modeling
- Limitations
This project is part of the Course Curriculum in Machine Learning at Frauenloop. The course was taught over the course of 3 months and presumed setting up a project in natural language processing (NLP).
Some of the concepts that were taught include:
+ types of NLP analyses,
+ connecting to an API and retrieving data,
+ data processing with spacy,
+ feature extraction with tf-idf
+ creating a pipeline with sklearn for modeling
+ evaluating model performance
I choose to focus on the sentiment analysis of tweets that were retrieved with the Twitter API Twython. The goal of my project is to predict the sentiment of tweets that are about Donald Trump. I made a list of 8 companies that openly support or are against Donald Trump, and I retrieved from the company's Twitter page 10.000 tweets. I assumed that the companies that openly support Donald Trump would tweet more positive tweets about him, and the companies that are against Donald Trump would tweet more negative tweets.
To set up your local environment before starting to work on the project, follow the steps:
-
Install venv library. Type in the terminal:
pip install venv -
Create virtual environment venv inside the root folder:
python -m venv venv -
Activate venv
Linux and MacOs:
source venv/bin/activateWindows:
venv/Scripts/Activate.ps1or cd to Scripts folder and typeactivate -
Upgrade pip:
python -m pip install --upgrade pip -
Install requirements:
pip install -r requirements.txt -
Install the spacy model:
python -m spacy download en_core_web_sm -
To request the data files, write me at: elenahirjoaba@gmail.com
The dataset is composed of 90.000 tweets retrieved from different companies'tweeter pages that openly support or are against Donald Trump. The list of companies is: Uline, Home_Depot, CNN, Taco_Bell, Bang_Energy, Patagonia, Microsoft, Merriam_Webster, Fox_News.
The scripts for file preparation can be found in src.data.file_preparation. The file contains functions to create a new raw_ and processed_file with headers, to write to .csv file, and to get relative file path.
The scripts for data retrieval can be found in src.data.data_retrieval. You can stream data specifying the keyword of the search, the id of the tweeter page, the number of minutes you want to stream, and the number of tweets you want.
The data is saved in data.raw.
The script for data preprocessing can be found in src.data.data_preprocessing. The processed .csv files can be found in data.processed. Stop words, urls, handles, punctuation are removed, and the emoji is transformed to a string according to the category it belongs to: EPOS or ENEG. The valence of the emoji is subjectively set by me. I separated positive and negative emojis in 2 lists.
The script for data labeling can be found in src.data.data_labeling. The .csv file with labeled data can be found in data.processed. I set 2 labels to evaluate the sentiment of the tweet: positive and negative. I first did a TF-IDF analysis on the corpus of words, then I manually categorized the most frequent words into a positive words list and a negative words list. I set the label of the tweet based on the max count of positive and negative words within the tweet. The tweets with no or equal number of positive/negative words in the tweet were discarded.
The script for data labeling can be found in src.features.feature_extraction. I applied a tf-idf analysis to retrieve feature.
Some limitations of the project and the analysis that was made includes: subjective evaluation of the sentiment of the emoji in the text and the valence of most common words in the corpora that were used for setting the label of the tweet.