TaxiDataPipeLine - Data Pipeline Project

Description
Build Status
Folder Structure
Getting Started
Prerequisites
Clone repository
Set-up development environment
How to run
How to Unit Test
How to check coverage
Data Source
Automated build setup
Future Enhancement Plan:

Description

TaxiDataPipeLine is an application to demonstrate creation of a simple data pipeline using Yellow Taxis trip data.

Build Status

	Build & Test
Windows x64
Linux x64

Folder Structure

docs - Project documentation
src - Python source code
test - Unit test

Getting Started

Follow these instructions to get the source code and run it on your local machine.

Prerequisites

You need Python 3.7.3 (Official download link) to run this project.

Clone repository

git clone https://github.com/write2sushma/TaxiDataPipeLine.git

Set-up development environment

Navigate to source folder

cd TaxiDataPipeLine

Create a virtual environment

In Linux OS

python3 -m venv env
source env\bin\activate

In Windows OS

python -m venv env
env\Scripts\activate

Install project dependencies

Project dependencies are listed in requirements.txt file. Use below command to install them -

pip3 install -r requirements.txt

If there is any issue in installing dask using requirements.txt file, use the below commands in command prompt/terminal window:

pip3 install “dask[complete]”

pip3 install dask distributed

How to run

Navigate to TaxiDataPipeLine\taxidata folder and run data_processor.py

python data_processor.py

How to Unit Test

Unit tests are written using Python's UnitTest library. Tests can be run using below command:

pytest

or 

python -m unittest test\test_data_processor.py

How to check coverage

Run below command to check code coverage:

python -m coverage run test\test_data_processor.py

And, then we can see coverage and can generate coverage report in html format

coverage report
coverage html

Data Source

Here is the list of data source urls used for creating data Pipe Line -

https://s3.amazonaws.com/nyc-tlc/trip+data/yellow_tripdata_2019-01.csv
https://s3.amazonaws.com/nyc-tlc/trip+data/yellow_tripdata_2019-02.csv
https://s3.amazonaws.com/nyc-tlc/trip+data/yellow_tripdata_2019-03.csv
https://s3.amazonaws.com/nyc-tlc/trip+data/yellow_tripdata_2019-04.csv
https://s3.amazonaws.com/nyc-tlc/trip+data/yellow_tripdata_2019-05.csv
https://s3.amazonaws.com/nyc-tlc/trip+data/yellow_tripdata_2019-06.csv

Automated build setup

Azure DevOp Pipeline is used to set and configure Automated build pipeline

Future Enhancement Plan:

• Optimize performance using dask scheduler to enable faster parallel processing.
    - This is already implemented in 'enhancements' feature branch.
• Scale pipeline to a multiple of the data size that does not fit any more to one machine using multinode clusters in cloud (e.g. AWS)
• Setup performance monitoring 
• Automate deployment using Azure DevOp Pipeline

saras152 / TaxiDataPipeLine