This repo contains all the notebooks used for sentimental analysis on the Sentiment140 dataset with PySpark. It was developed part of an end-of-term project for 8INF919 : Machine Learning for Big Data at UQAC in collaboration with Thomas Sirvent.
You can find in the repo the LaTeX report and the presentation slides associated to this project (in french). If you'd like to read english explanations check it out my website.
We worked with the following models :
- Logistic Regression
- Support Vector Machines (Linear Kernel)
- Naive Bayes
- Random Forest
- Decision Tree
- Hashing TF-IDF
- Count Vectorizer TF-IDF
- ChisQSelector
- 1-Gram, 2-Gram, 3-Gram
In the notebooks directory, you'll find the a Python file called "cluster_logistic_job.py" if you are curious and you want to see how we ran our models in the Cloud.
Another part of this project was to implement an ETL Pipeline with Live Sentiment Analysis using our pre-trained model, Spark Streaming, Apache Kafka and Docker. The repository for this part is available here.