This is an example on integrating FastAPI and additional libraries with OpenTelemetry and common tracing tools. The goal is to create more realistic scenarios (if one could say that about a fantasy setting) in terms of an application that goes beyond 'simple CRUD' and can demonstrate the value of using distributed tracing in the development day to day. It should be disclaimed and pointed out that this does not purport to be in any way a 'go-to' architecture - please don't use it as such! It is intended to examine different architectural problems and how we can use observability to solve them.
This example is based on a great repo by @grillazz from which it was also original forked.
The fully instrumented stack includes:
- SQLAlchemy ORM
- PostgreSQL via asyncpg
- Async HTTP via httpx we will use it to connect to a mock site created via mockapi
- RabbitMQ queuing via pika
From the tracing stack side we'll use:
- Jaeger for distributed tracing
- Prometheus for metrics (TBD still WIP)
- Digma for getting observability insights back into the code.
- Docker and Docker Compose (not a hard requirement, but makes woking with this example easier)
- Python 3.8+
- vsCode - For some aspects of developer observability
- Clone or download the repo
docker compose -f ./observability/digma-cf/docker-compose.digma.yml up -d
docker compose -f ./observability/tracing/docker-compose.trace.yml up -d
2. Start the Gringotts API application, including all services and background 'Goblin worker' process
docker compose --profile standalone -f docker-compose.yml -f docker-compose.override.standalone.yml up -d
This will import in some Wizards from a data source I found online, to make it a little easier to work with the API
docker compose -f ./docker-compose.seed.yml up --attach gt-seed-data
Go to the API documentation page at http://localhost:8283/docs and login using the authenticate button.
Since we populated some data you can use the following details:
username: hpotter password: griffindoor
Notice that it takes some time to process, we'll look into that in a sec.
Install the Digma extension on your local IDE. We'll use Digma to demonstrate continuous feedback.
- Open http://localhost:16686 to access the Jaege instance. By now you should already have some data to look at.
- Upon installing the Digma extension open your vscode and select the Digma icon to see information in the context of the code.
Create a separate python env for the worker and gringotts folders. For each environment install the local requirements.txt file.
For example, using virtualenv.
cd gringotts
python -m venv ./venv
source ./venv/bin/activate
pip install -r requirements.txt
I had an issue with this requirement on my local development laptop. I had to manually install openssl and then install via pip after setting these environment variables:
LDFLAGS="-I/opt/homebrew/opt/openssl@3/include -L/opt/homebrew/opt/openssl@3/lib" pip install psycopg2-binary
The solution to this known issue is posted in several places including here
To simply run the backend services (Postgres, RabbitMQ) to develop locally from the IDE, run:
docker compose --profile develop -f docker-compose.yml -f docker-compose.override.develop.yml up -d
To run the tests via docker compose, use the
docker compose --profile test -f docker-compose.yml -f docker-compose.override.test.yml up --abort-on-container-exit --attach gt-vault-api
To run the tests with some seeded data to see how the system behaves with data in place, use:
PYTEST_ARGUMENTS="--seed-data true" docker compose --profile test -f docker-compose.yml -f docker-compose.override.test.yml up --abort-on-container-exit --attach gt-vault-api --abort-on-container-exit
