These are ready in Pycharm run configurations.
DB_TYPE=sql
CELERY_RESULT_BACKEND=redis://localhost:6379/0
CELERY_BROKER_URL=redis://localhost:6379/0
Prerequisites
- python 3.11
- poetry
- docker
- postgresql (
brew install postgresql
)
Installation
poetry install
docker compose up
poetry run streamlit run ui/dashboard.py
Task queue is part of docker services, but can be run separately as well
poetry run celery --app backend.worker worker --loglevel INFO
SqlAlchemy
andpymongo
libraries are used for database driver.Streamlit
is used as the main user facing application. This library allows developers to create a reactive server in pure python. UI elements are defined in python files and generate html when the client connects, while the database connections are kept on server side.Celery
- We would like to keep long-running processes independent of the user facing application server in order to have a more robust and reliable service. The database migration is dispatched as a celery task which will run independently of the application server.
- Database operations and schemas
- celery tasks
All the files in this package are ui components, and are rendered when the user visits the corresponding page (dashboard.py
is the homepage, and pages
folder has the rest of the website pages). Connections to the database and celery workers are handled in server_connections
following containers are part of the full deployment
- postgres
- mongo
- redis
- our package - entrypoint
celery backend/worker.py
for running celery workers - our package - entrypoint
streamlit ui/dashboard.py
for running the application
poetry run export_schemas
I also set up to do it from alembic but the result is same. Still will document the process
alembic init alembic
Change alembic configuration in env.py
:
from backend.sql.models import Base
# find this line and change to
# target_metadata = None
target_metadata = Base.metadata
Change sqlalchemy url in alembic.ini
to postgresql://postgres:postgres@localhost/backend
alembic revision --autogenerate -m "initialise schemas"
copy revision number from the output
alembic upgrade $REVISION_NUMBER --sql > create_table.sql
You will need to delete alembic/versions
to regenerate it
- track task progress https://pypi.org/project/celery-progress/