ibeketov / EDS_2

Disaster Response Pipeline Project

Portfolio project to showcase Data Engineering skills including ETL and ML Pipeline preparation, utilising model in a web app, and data visualisation.

Project Components

1. ETL Pipeline The Python script, process_data.py:

   • Loads the messages and categories datasets
   • Merges the two datasets
   • Cleans the data
   • Stores it in a SQLite database

2. ML Pipeline The Python script, train_classifier.py:

   • Loads data from the SQLite database
   • Splits the dataset into training and test sets
   • Builds a text processing and machine learning pipeline
   • Trains and tunes a model using GridSearchCV
   • Outputs results on the test set
   • Exports the final model as a pickle file

3. Flask Web App

File Descriptions

There are three main foleders:

1. data
   • disaster_categories.csv: dataset including all the categories
   • disaster_messages.csv: dataset including all the messages
   • process_data.py: ETL pipeline scripts to read, clean, and save data into a database
   • DisasterResponse.db: output of the ETL pipeline, i.e. SQLite database containing messages and categories data
2. models
   • train_classifier.py: machine learning pipeline scripts to train and export a classifier
   • DecisionTreeClassifier.pkl: output of the machine learning pipeline, i.e. a trained classifer
3. app
   • run.py: Flask file to run the web application
   • templates contains html file for the web applicatin

Results

1. An ETL pipleline was built to read data from two csv files, clean data, and save data into a SQLite database.
2. A machine learning pipepline was developed to train a classifier to performs multi-output classification on the 36 categories in the dataset.
3. A Flask app was created to show data visualization and classify the message that user enters on the web page.

Requirements

Please check the requirements.txt for complete information about working environment.

Please use pip install -r requirements.txt to run it

Instructions:

1. Run the following commands in the project's root directory to set up your database and model.

   • To run ETL pipeline that cleans data and stores in database python data/process_data.py data/disaster_messages.csv data/disaster_categories.csv data/DisasterResponse.db
   • To run ML pipeline that trains classifier and saves python models/train_classifier.py data/DisasterResponse.db models/classifier.pkl

2. Run the following command in the app's directory to run your web app. python run.py

3. Go to http://0.0.0.0:3001/