datahappy1 / czech_language_sentiment_analyzer

Czech sentiment analyzer

10000 ft. Overview

Data Collection

56k Czech movie reviews were collected using the /data_preparation/data_collector_movie_review_scraper.py multithreaded HTML scraping module. These reviews were scrubbed using langdetect module to remove reviews written in Slovak language. This dataset was also scrubbed against a collection of Czech stopwords. To have the data balanced with the same amount of negative and positive reviews, the final dataset had to be reduced to 11.5k positive and 11.5k negative reviews. Collected data was also stemmed before training the models.

ML Models

From Scikit-Learn Python library, Naive Bayes, Logistic regression and Support Vector Machine ML models were used for training and testing data for text sentiment analysis. The scripts for training and testing are located here:

• /ml_models/logistic_regression
• /ml_models/naive_bayes
• /ml_models/support_vector_machine

The overall sentiment score for the specified text input is calculated as a weighted average based on the precision score accuracy of these 3 model predictions.

Flask web application

The Flask web application is currently hosted at https://czester.herokuapp.com, source code can be found in this location /flask_webapp/. This application backend is written in Python using the Flask framework and Bootstrap for the templates styling. This app also provides the users with a simple API. The stats module is a result of an integration between Chart.js and Flask where the statistics data persistence layer can be either Sqlite3 or Heroku Postgres. If you provide this app with a environment variable named DATABASE_URL containing the Heroku Postgres DB URL like postgres://YourPostgresUrl, then remote Heroku Postgres will be used, otherwise local Sqlite3 db instance will be used.

Input text dataflow diagram:

How to run this Flask App from local environment

1. create and activate a standard Python virtual or pipenv environment
   
2. pip3 install the requirements from requirements.txt
   
3. set the working directory for instance to the path where you cloned this repo (Make sure it's the path where the Heroku Procfile file is located)

TODOs / Future ideas

• Remove reviews written in Slovak language ✓
• Verify input text is written in Czech language ✓
• Add Flask web app tests ✓
• Add Czech word stemmatizer module ✓
• Ensembling instead of weighted model precision average for overall sentiment
• Redis could replace Sqlite3 / Postgres
• Migrate from Heroku to AWS

Useful links

• Scikit-Learn working with text data
• Ensembling with Scikit-Learn
• Ensembling models
• Langdetect PyPi project homepage
• Chart.js homepage
• Deploying Flask to Heroku tutorial