Django + Kafka + Celery π
pip install confluent-kafka
- Download confluent platform from: https://www.confluent.io/download/ and unzip the file.
export CONFLUENT_HOME=<path-to-confluent>
export PATH=$PATH:$CONFLUENT_HOME/bin
$CONFLUENT_HOME/bin/confluent-hub install \ --no-prompt confluentinc/kafka-connect-datagen:latest
The output would be:
Running in a "--no-prompt" mode
...
Completed
- Start the service:
confluent local start
The output should resemble:
Starting Zookeeper
Zookeeper is [UP]
Starting Kafka
Kafka is [UP]
Starting Schema Registry
Schema Registry is [UP]
Starting Kafka REST
Kafka REST is [UP]
Starting Connect
Connect is [UP]
Starting KSQL Server
KSQL Server is [UP]
Starting Control Center
Control Center is [UP]
- Stop services:
confluent local stop
Or set-up Kafka and Zookeeper Separately
- Simply put, Kafka is a distributed publish-subscribe messaging system that maintains feeds of messages in partitioned and replicated topics.
- In the simplest way there are three players in the Kafka ecosystem: producers, topics (run by brokers) and consumers.
brew install zookeeper
- Where is installation directory of zookeeper :
/usr/local/Cellar/zookeeper
- In foreground
zkServer start
- In background
brew services start zookeeper
brew install kafka
- Where is installation directory of Kafka :
/usr/local/Cellar/kafka
- In foreground
brew services start kafka
- In background
zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties & kafka-server-start /usr/local/etc/kafka/server.properties
For details on how to set-up a django project with best practices: https://pyblog.xyz/django-initial-setup/
-
pip install amqp
-
pip install celery==4.4.0
(Note: Celery 5.0 is not compatible with django-celery-beat) -
pip install django-celery-beat
-
Start worker and beat
celery -A django_kafka beat -l INFO -S django
-
Start RabbitMQ
brew services start rabbitmq
INSTALLED_APPS = [
...,
'django_celery_beat',
]
- Run migrations:
python manage.py migrate django_celery_beat
Note: In this project, the same Django project is the Producer and Consumer, but you can choose you have a standalone consumer.