Simple Python MQTT Data Logger
This software uses the Python logger to create a logfile for all messages for all topics to which this MQTT client has subscribed.
Note: by default it will only log changed messages. This is for sensors that send out their state a regular intervals but that state doesn't change The program is run from the command line
You can subscribe to multiple topics.
You need to provide the script with:
- List of topics to monitor
- broker name and port
- username and password if needed.
- base log directory and number of logs have defaults
Valid command line Options:
--help <help>
-h <broker>
-b <broker>
-p <port>
-t <topic>
-q <QOS>
-v <verbose>
-d logging debug
-n <Client ID or Name>
-u Username
-P Password
-s <store all data>\
-l <log directory default= mlogs>
-r <number of records default=100>\
-f <number of log files default= unlimited"
-j <JSON>
Install dependencies
You need the paho mqtt library. Using pip3:
pip install paho-mqtt
Example Usage:
You will always need to specify the broker name or IP address and the topics to log.
Note: you may not need to use the python prefix or may need to use python3 mqtt_data_logger.py (Linux)
Specify broker and topics
python mqtt_data_logger.py -b 192.168.1.157 -t sensors/#
Specify broker and multiple topics
python mqtt_data_logger.py -b 192.168.1.157 -t sensors/# -t home/#
Log All Data (plain text format):
python mqtt_data_logger.py b 192.168.1.157 -t sensors/# -s
Log All Data (JSON format):
python mqtt_data_logger.py b 192.168.1.157 -t sensors/# -s -j
Specify the client name used by the logger
python mqtt_data_logger.py b 192.168.1.157 -t sensors/# -n data-logger
Specify the log directory
python mqtt_data_logger.py b 192.168.1.157 -t sensors/# -l mylogs
Logger Class
The class is implemented in a module called m_logger.py (message logger).
To create an instance you need to supply three parameters:
-
The log directory- defaults to mlogs
-
Number of records to log per log- defaults to 5000
-
Number of logs. 0 for no limit.- defaults to 0
log=m_logger(log_dir="logs",log_recs=5000,number_logs=0):
The logger creates the log files in the directory using the current date and time for the directory names.
The format is month-day-hour-minute e.g.
You can log data either in plain text format or JSON format.
To log data either in plain text then use the
log_data(data) method.
To log data as JSON encoded data call the
log_json(data) method.
Both method takes a single parameter containing the data to log as a string, list or dictionary..
e.g.
log.log_data(data)
or
log.log_json(data)
The log file will contain the data as plain text or JSON encoded data strings each on a newline.
The logger will return True if successful and False if not.
To prevent loss of data in the case of computer failure the logs are continuously flushed to disk .
Read more about this application here:
http://www.steves-internet-guide.com/simple-python-mqtt-data-logger/