This app implements a producer component of classic Producer-Consumer Architecture. It exposes a web API which accepts CSV files. CSV files are passed to a celery task. This task reads all the data from file based on specified format and push everything to attached broker. For this application the choice of broker is limited to the ones which support named queues creation.
The following steps will get you a copy of app(single instance) on you local system:
git clone https://github.com/sohaibfarooqi/producer.git
cd producer
virtualenv -p python3 .venv
source .venv/bin/activate
pip install -r requirements/dev.txt
set environemnt variable CELERY_BROKER_URL=amqp://myuser:mypassword@localhost:5672/myvhost
export FLASK_APP=producer
Run the following two commands in two seperate shells
flask run
celery -A producer worker -l info
At this point the app is ready to accept new files. Use the following script to test:
Install requests package using pip install requests
import requests
files = {'file': open('test.csv', 'r')}
requests.post('http://localhost:5000/upload', files=files)
NOTE By default the producer will send messages to test
queue and expect consumer to
be a celery task named consumer
.
Follow these commands to run tests and generate coverage reports
pip install -r requirements/test.txt
pytest
pytest --cov=producer tests/
- Celery Distributed queue management.
- Flask WSGI server.
- Isort Sorting and arranging imports.
- Autopep8 Code styling to conform with PEP8
- Pytest Running test cases.
Github pre-commit hooks can be ver useful to automate things like code formating, running linters and checking for missing migrations. Use following commands to enable them:
- Run
pre-commit install
to enable the hook into your git repo. The hook will run automatically for each commit. - Run
git commit -m "Your message" -n
to skip the hook if you need.