Json2mqtt is a little python daemon that retrieves data from json api's and sends selected fields to an MQTT broker.
It can be used to retrieve data from any http api that returns json.
Write a (bunch of) schema file(s), feed it to the daemon by files or over mqtt and every interval you receive the returns on a mqtt topic.
Using the schemas you can give fine-grained instructions to the daemon which elements you want to publish at which interval
For each schema, a new timer is started in its own thread.
They remain in sleep state until the next interval, and keep doing this until their count value is reached (or indefinitely).
When a timer triggers a new data retrieval, the requested fields and some metrics about the request are sent to the mqtt broker.
All schemas, in the schemas_dir
directory, as configured in settings.yaml
are loaded at start-up unless explicitly disabled
setting "enabled": false
in the schema definition.
There are some predefined schemas that can be used
in the schemas
directory in this repository,
that you can use to poll a rooted Toon.
A schema is a json definition containing instructions for the daemon on which fields to publish to MQTT
You can write your own schemas and feed them to this service as a file or over MQTT.
A schema consists of 4 main (required) elements:
-
name
- The name of the schema -
url
- - The url to retrieve json data from -
interval
- The time in seconds between retrievals -
fields
- A list of dictionaries each containing a path and a type element defining fields that are to be expected in the retrieved json. (more info below)
Additional optional elements are:
-
topic
- Override the topic where to send the data to. (The topic is appended with the value of thename
of the schema) -
count
- How many times the crawl should be performed. Default is-1
, which keeps repeating the request until the timer is stopped, waiting to be started again -
timeout
- The timeout for the http requests (Default is 10s) -
headers
- A list of key value pairs with additional headers (can be used for host, auth, user-agent etc) -
enabled
- Explicitly enable or disable the schema at startup (The schema needs to exist on disk to be loaded at startup)
A field
element consists of 2 required fields:
path
- The jmespath of the value to send to mqtttype
- The type of data in this field
Optional fields are:
cast
- Cast the currenttype
to another type, useful for "number" or "true" and other returned strings. Allowed values are equal to thetype
element, but with a bit of common sense, and a bit of python added.
The types available:
String
Integer
Float
Boolean
None
List
Dictionary
Example: retrieve the module version data from the toon and send over MQTT
{
"name": "module_version",
"url": "http:///toon.local/happ_thermstat?action=getModuleVersion",
"interval": 3600,
"timeout": 30,
"headers": [
{
"key": "User-Agent",
"value": "Json2MQTT"
},
{
"key": "Authorization",
"value": "Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ=="
}
],
"fields": {
"mmb": {
"type": "String",
"cast": "Boolean",
"path": "mmb"
},
"version": {
"type": "String",
"path": "version"
}
}
}
The topics used to publish to in this example are:
home/json2mqtt/module_version/mmb
home/json2mqtt/module_version/version
Or you can override the topics using the topic
field.
git clone https://github.com/fliphess/json2mqtt.git && cd json2mqtt
Create a virtualenv and install json2mqtt
and its requirements:
python3 -m venv venv
source venv/bin/activate
pip install .
A config file is created on first run if nonexistent:
json2mqtt --settings settings.yaml --init
vim settings.yaml
json2mqtt --config settings -vvv
docker build -t json2mqtt .
docker run \
-ti \
-v "/tmp:/cfg" \
--rm json2mqtt \
--config /cfg/settings.yaml
mv /tmp/settings.yaml .
vim settings.yaml
docker run \
--rm \
-ti \
-v "$(pwd)/settings.yaml:/opt/json2mqtt/settings.yaml" \
-v "$(pwd)/schemas:/opt/json2mqtt/schemas" \
json2mqtt --config settings.yaml -vvv
The config file will be created on first run if it does not exist. It will fill up the required fields with the default values. This will not suffice to start the daemon, as you need to configure your broker first.
A full configuration file contains:
mqtt_host: "some.broker"
mqtt_port: 1883
mqtt_username: "username"
mqtt_password: "password"
mqtt_topic: home/toon2mqtt
mqtt_ssl: true
mqtt_cert: "/etc/ssl/cert.pem"
schema_dir: ./schemas
You can control the daemon over MQTT.
You can start and stop timers, add, remove and reload schemas.
The topics to which the retrieved data and response metrics are send:
home/json2mqtt/<schema name>/<key> # Retrieved json keys from api
home/json2mqtt/<schema name>/request/success # If the last request succeeded
home/json2mqtt/<schema name>/request/status_code # The status code of the last request
home/json2mqtt/<schema name>/request/reason # The reason of the last request
home/json2mqtt/<schema name>/request/url # The full url of the request
The topics to publish to, to create, list and manipulate the schemas
home/json2mqtt/command/schema/add # Add a json schema from the mqtt payload
# Input: a json schema
home/json2mqtt/command/schema/add_file # Add a json schema from a file that is present on disk
# Input: a filename of a file in the schema_dir
home/json2mqtt/command/schema/remove # Remove/Disable a schema
# Input: the name of the schema to remove (See: `schema/list`)
home/json2mqtt/command/schema/list # List all json schemas
# No input required, an empty string or a 0 suffices.
home/json2mqtt/command/schema/import # Import all schemas from disk
# No input required, an empty string or a 0 suffices.
home/json2mqtt/command/schema/dump # Write all schemas to disk
# No input required, an empty string or a 0 suffices.
Schemas can be manipulated, loaded and written to disk. They are used by the scheduler, but not automatically renewed. To update the timers that use the schemas, you additionally need to reload the scheduler task(s)
Timers can be controlled separately from the schemas that are used to instruct what to crawl and how often.
home/json2mqtt/command/scheduler/list # List all timers
# No input required, an empty string or a 0 suffices.
home/json2mqtt/command/scheduler/stop # Stop all timers
# No input required, an empty string or a 0 suffices.
home/json2mqtt/command/scheduler/start # Start all (not running) timers
# No input required, an empty string or a 0 suffices.
home/json2mqtt/command/scheduler/add_timer # Add a timer using a json schema, do not save anything to schemas,
# just start the timer (single use, will be gone after a restart)
home/json2mqtt/command/scheduler/remove_timer # Remove a timer
home/json2mqtt/command/scheduler/start_timer # Start a stopped timer
home/json2mqtt/command/scheduler/pause_timer # Stop a running timer
All commands return their output to home/json2mqtt/talkback
Casting string "1" and "0" values to a boolean is tricky:
In [1]: bool("0")
Out[1]: True
Instead, don't do a cast at all and use the 1
and 0
strings directly.
Due to the way jmespath for python works, keys containing a dot, need to be contained by escaped double quotes:
{
"fields": {
"electricity_delivered_lt_flow": {
"type": "String",
"path": "\"dev_2.6\".CurrentElectricityFlow"
},
}
}