Natural Language Processing with Disaster Tweets
Introduction
This project features an implementation of an NLP pipeline for the disaster tweets Kaggle competition using the Brane framework. The implementation is divided into the following Brane packages which can be imported individually and used in other workflows: compute, visualization, and utils.
computeexposes utilities for preprocessing data, training a classifier, and generating a valid submission file for the challenge.visualizationprovides functions to generate plots and charts based on the dataset.utilscontains generic utility functions and specifically allows for downloading the dataset at runtime.
We also include a github.yml specification which defines an OpenAPI container that exposes a function to download arbitrary files from GitHub repositories.
Build
Each package can be individually imported with the following command:
brane import marinoandrea/disaster-tweets-brane -c packages/<PACKAGE_NAME>However, we also provide a shell script for convenience. The user can clone the repository and simply run ./build.sh all to build all of our packages. Additionally, you also can run the following commands to build a specific package.
# build the computation package
./build.sh compute
# build the visualization package
./build.sh visualization
# build the utils package
./build.sh utilsOf course, you can always navigate to the package directory and run the following command to build the brane package.
brane build container.yml
Run
Our pipeline implementation can be executed locally or on a multi-node Kubernetes cluster by simply running the following command in the root folder of the project:
brane run -d <DFS_FOLDER> pipeline.bsThe following picture shows an example that our package uses the pipeline.bs to run the whole pipeline in the Kubernetes cluster.
NOTE: Brane may print some warnings about serialization issues to the console. However, the pipeline can run till the end without issues.