This is a demo project to represent a way to implement Python microservices with Kafka and Faust library.
01_infrastructure - contains basic infrastructure described in docker-compose.yml file. Contains all infrastructure services such as:
- Kafka
- Zookeeper
- Postgres
- PgAdmin
- Prometheus
- Grafana
In .env file described all versions and ports required for the docker-compose.yml The grafana folder contains preconfigured dashboards and datasources required for this docker-compose.yml The prometheus folder contains preconfigured services to collect metrics by the prometheus service, required for this docker-compose.yml
02_demo_server - contains demo server app and Dockerfile required to build it as docker image. The demo server produce random test data needed for demo.
03_data_requester - contains data request microservice app and Dockerfile required to build it as docker image. The data requester service request data from demo server and send it to a Kafka.
04_data_processor - contains data processor microservice app and Dockerfile required to build it as docker image. The data processor service read messages received from data requester via Kafka. Split data and upload to Kafka.
05_data_aggregator - contains data aggregator microservice app and Dockerfile required to build it as docker image. The data aggregator service read messages received from data processor via Kafka. And calculate average value for last 10 responses.
06_db_loader - contains db loader microservice app and Dockerfile required to build it as docker image. The db loader service read messages from data processor and data aggregator services and store this data to Postgres DB.
07_api_gateway - contains API gateway microservice app and Dockerfile required to build it as docker image. The api gateway service provide HTTP endpoints to request data from Postgres DB.
docker-compose.yml - contains all services and infrastructure in one.
- Python 3.8+
- Docker 20.10.8
- Docker-compose 1.29.2
- asyncio==3.4.3
- aiohttp==3.7.4.post0
- uvloop==0.15.3
- fastapi==0.68.0
- uvicorn[standard]==0.14.0
- starlette==0.14.2
- starlette-exporter==0.10.0
- configloader==1.0.1
- prometheus-client===0.11.0
- PyYAML==5.4.1
- asyncpg==0.23.0
- SQLAlchemy==1.4.0
- faust[rocksdb, fast, uvloop]==1.10.4