(requirements: localhost needs docker
and docker-compose (v1)
to be installed and running)
N.B: if you have docker compose (v2)
installed instead, follow step 2 in the section on running app on ec2 instead
- clone repo
git clone https://github.com/hqtan/mqtt-exercise-python.git`
cd mqtt-exercise-python
- run
docker-compose
to see the calculated averages N.B:
- the first run may take a while, as it pulls and builds the docker images
docker-compose up --build
console out should look something like this:
stats_subscriber | avg_1min avg_5min avg_30min
stats_subscriber | ---------- ---------- -----------
stats_subscriber | 46 45.7 47.1778
stats_subscriber | avg_1min avg_5min avg_30min
stats_subscriber | ---------- ---------- -----------
stats_subscriber | 56 49 48.2391
stats_subscriber | avg_1min avg_5min avg_30min
stats_subscriber | ---------- ---------- -----------
stats_subscriber | 60.4 50.6667 49
...
to run app in 'verbose' mode (to see the console output of mosquitto
, publisher
and subscriber
container), run this command:
docker-compose -f docker-compose-debug.yml up --build
- ssh to ec2 instance
docker
anddocker-compose
has already been installed, and the repo has already been cloned.cd
to repo and rundocker compose
docker compose up --build --attach stats_subscriber
to run app in 'verbose' mode (to see the console output of mosquitto
, publisher
and subscriber
container), run this command:
docker compose up --build
- all published messages must be received by subscriber(s), ie: no dropped messages
- no auth and TLS used: publisher and subscriber running in secure environment
- timestamp not timezone sensitive (in UTC time)
- messages can be stored in sqlite db
- all the processes can be containerised
- All processes are containerised. They are separated into 4 containers:
mosquitto
(broker) - runs the mosquitto mqtt brokerpublisher
- runs thepublish.py
python process. Publishes a number between 1-100 at random intervals between 1-30 seconds to 'DATA' topicsubscriber
- runs thesubscribe.py
python process. It does a number of things:- subscribes to 'DATA' topic
- creates a
messages
db table (if it doesn't exist), and inserts messages from 'DATA' topic into table - cleans messages from table that are older than 35 minutes
- gets the calculated averages from table, and publishes it to 'STATS' topic
stats_subscriber
- runs thestats_pubsub.py
python process. It subscribes from the 'STATS' topic, and prints the averages in a tabular layout to console.
- containers are 'orchestrated' using
docker compose
. Start order is:mosquitto
container starts up firstsubscriber
container starts next, and connects to broker, subscribed to 'DATA' topicstats_subscriber
container starts next, and connects to broker, subscribed to 'STATS' topic- finally the
publisher
container starts, when the broker and subscriber processes are up.
- the random numbers and associated timestamps (in epoch seconds) published on 'DATA' topic are stored in a sqlite3 db.
- only the last 35 minutes of data are kept. The rest are deleted. This 'cleaning' process is done in the
subscriber
process'on_message
callback.
- only the last 35 minutes of data are kept. The rest are deleted. This 'cleaning' process is done in the
- Averages calculated using SQL
AVG
function, and then published bysubscriber
container to 'STATS' topic. The stats are not saved into sqlite db. .env
for environment variables used by python processes. The environment variables used are:BROKER
- the hostname for MQTT broker. Defaults tolocalhost
. In the docker containers, this has been set tomosquitto
(the name of container running the mosquitto service)DB_FILE
- the name of the file containing the sqlite db. Defaults totmp/test.db
STATS_TOPIC
- the name of the topic containing the statsTOPIC
- the name of the topic containing the data