Documentation for serial protocol
requeijaum opened this issue · comments
I can't find the message format for using the bridge correctly.
I was successfully using another project but it's bridge can't receive MQTT messages from Serial (reverse way), only publish to a broker.
Here is a quick log from your application:
vd@vd1:~/downloads/2mqtt-1.3-linux-arm64$ ./2mqtt
2023-02-17T11:53:01.168Z info start/logger.go:14 welcome to the 2mqtt adapter server :)
2023-02-17T11:53:01.168Z info start/logger.go:15 server detail {"version": {"version":"1.3","gitCommit":"0667010e106bc840db78c3b8d157498d8e116856","buildDate":"2022-06-08T10:20:20+00:00","goLang":"go1.18.2","platform":"linux","arch":"arm64"}, "loggerConfig": {"Mode":"development","Encoding":"console","Level":"debug"}}
2023-02-17T11:53:01.169Z debug mqtt/device.go:60 mqtt config {"adapterName": "adapter1", "config": {"Name":"","Broker":"tcp://127.0.0.1:1883","InsecureSkipVerify":true,"Username":"","Subscribe":"vallydroid/#","Publish":"vallydroid/","QoS":0,"TransmitPreDelay":"0s","ReconnectDelay":"5s"}}
2023-02-17T11:53:01.170Z debug mqtt/device.go:94 mqtt client connected successfully {"adapterName": "adapter1", "timeTaken": "1.579829ms", "clientConfig": {"Name":"","Broker":"tcp://127.0.0.1:1883","InsecureSkipVerify":true,"Username":"","Subscribe":"vallydroid/#","Publish":"vallydroid/","QoS":0,"TransmitPreDelay":"0s","ReconnectDelay":"5s"}}
2023-02-17T11:53:01.170Z info adapter/service.go:238 connected to the mqtt broker {"adapterName": "adapter1"}
2023-02-17T11:53:01.170Z debug mqtt/device.go:103 mqtt connection success {"adapterName": "adapter1"}
2023-02-17T11:53:01.171Z debug serial/device.go:58 generated config {"config": {"Port":"/dev/ttyUSB0","BaudRate":115200,"MessageSplitter":10,"TransmitPreDelay":"10ms"}}
2023-02-17T11:53:01.171Z info serial/device.go:62 opening a serial port {"adapterName": "adapter1", "port": "/dev/ttyUSB0"}
2023-02-17T11:53:01.171Z debug mqtt/device.go:186 subscribed a topic {"adapterName": "adapter1", "topic": "vallydroid/#"}
2023-02-17T11:53:01.177Z info adapter/service.go:270 connected to the source device {"adapterName": "adapter1"}
2023-02-17T11:53:01.177Z info start/start.go:26 services started {"timeTaken": "10.175326ms"}
2023-02-17T11:53:08.511Z debug adapter/service.go:172 received a message from source device {"message": "{data:[1,\"vallydroid/sensors/24AE0525BF58/send\",\"JK4FJb9YAQ0BFwEH1SoCnwDKAAAQsBA=\"]\r, others:map[], timestamp:2023-02-17 11:53:08.511578315 +0000 GMT m=+7.347275376"}
2023-02-17T11:53:09.178Z debug mqtt/device.go:134 about to send a message {"adapterName": "adapter1", "message": "{data:[1,\"vallydroid/sensors/24AE0525BF58/send\",\"JK4FJb9YAQ0BFwEH1SoCnwDKAAAQsBA=\"]\r, others:map[mqtt_qos:0 mqtt_topic:], timestamp:2023-02-17 11:53:08.511578315 +0000 GMT m=+7.347275376"}
2023-02-17T11:53:09.181Z debug adapter/service.go:162 received a mqtt message {"message": "{data:[1,\"vallydroid/sensors/24AE0525BF58/send\",\"JK4FJb9YAQ0BFwEH1SoCnwDKAAAQsBA=\"]\r, others:map[mqtt_qos:0 mqtt_topic:vallydroid/], timestamp:2023-02-17 11:53:09.181021406 +0000 GMT m=+8.016718560"}
2023-02-17T11:53:09.194Z debug adapter/service.go:162 received a mqtt message {"message": "{data:[1,\"vallydroid/sensors/24AE0525BF58/send\",\"JK4FJb9YAQ0BFwEH1SoCnwDKAAAQsBA=\"]\r, others:map[mqtt_qos:0 mqtt_topic:vallydroid/], timestamp:2023-02-17 11:53:09.19393239 +0000 GMT m=+8.029629562"}
2023-02-17T11:53:09.992Z debug adapter/service.go:172 received a message from source device {"message": "{data:[1,\"vallydroid/sensors/E04066F23A08/send\",\"4EBm8joIAQ0BGgEM1SkCcwC/AAAQrhA=\"]\r, others:map[], timestamp:2023-02-17 11:53:09.992556235 +0000 GMT m=+8.828253407"}
2023-02-17T11:53:10.178Z debug adapter/service.go:148 posting a message to source device {"message": "{data:[1,\"vallydroid/sensors/24AE0525BF58/send\",\"JK4FJb9YAQ0BFwEH1SoCnwDKAAAQsBA=\"]\r, others:map[mqtt_qos:0 mqtt_topic:vallydroid/], timestamp:2023-02-17 11:53:09.181021406 +0000 GMT m=+8.016718560"}
2023-02-17T11:53:10.178Z debug mqtt/device.go:134 about to send a message {"adapterName": "adapter1", "message": "{data:[1,\"vallydroid/sensors/E04066F23A08/send\",\"4EBm8joIAQ0BGgEM1SkCcwC/AAAQrhA=\"]\r, others:map[mqtt_qos:0 mqtt_topic:], timestamp:2023-02-17 11:53:09.992556235 +0000 GMT m=+8.828253407"}
2023-02-17T11:53:10.179Z debug adapter/service.go:162 received a mqtt message {"message": "{data:[1,\"vallydroid/sensors/E04066F23A08/send\",\"4EBm8joIAQ0BGgEM1SkCcwC/AAAQrhA=\"]\r, others:map[mqtt_qos:0 mqtt_topic:vallydroid/], timestamp:2023-02-17 11:53:10.179059469 +0000 GMT m=+9.014756512"}
2023-02-17T11:53:10.186Z debug adapter/service.go:162 received a mqtt message {"message": "{data:[1,\"vallydroid/sensors/E04066F23A08/send\",\"4EBm8joIAQ0BGgEM1SkCcwC/AAAQrhA=\"]\r, others:map[mqtt_qos:0 mqtt_topic:vallydroid/], timestamp:2023-02-17 11:53:10.186665782 +0000 GMT m=+9.022362825"}
2023-02-17T11:53:10.188Z debug adapter/service.go:148 posting a message to source device {"message": "{data:[1,\"vallydroid/sensors/24AE0525BF58/send\",\"JK4FJb9YAQ0BFwEH1SoCnwDKAAAQsBA=\"]\r, others:map[mqtt_qos:0 mqtt_topic:vallydroid/], timestamp:2023-02-17 11:53:09.19393239 +0000 GMT m=+8.029629562"}
2023-02-17T11:53:10.199Z debug adapter/service.go:148 posting a message to source device {"message": "{data:[1,\"vallydroid/sensors/E04066F23A08/send\",\"4EBm8joIAQ0BGgEM1SkCcwC/AAAQrhA=\"]\r, others:map[mqtt_qos:0 mqtt_topic:vallydroid/], timestamp:2023-02-17 11:53:10.179059469 +0000 GMT m=+9.014756512"}
2023-02-17T11:53:10.209Z debug adapter/service.go:148 posting a message to source device {"message": "{data:[1,\"vallydroid/sensors/E04066F23A08/send\",\"4EBm8joIAQ0BGgEM1SkCcwC/AAAQrhA=\"]\r, others:map[mqtt_qos:0 mqtt_topic:vallydroid/], timestamp:2023-02-17 11:53:10.186665782 +0000 GMT m=+9.022362825"}
2023-02-17T11:53:30.259Z debug adapter/service.go:172 received a message from source device {"message": "{data:[1,\"vallydroid/sensors/184166F23A08/send\",\"GEFm8joIAQ8BHQEL1SoCiwDIAAAQuBA=\"]\r, others:map[], timestamp:2023-02-17 11:53:30.25971624 +0000 GMT m=+29.095413264"}
2023-02-17T11:53:31.178Z debug mqtt/device.go:134 about to send a message {"adapterName": "adapter1", "message": "{data:[1,\"vallydroid/sensors/184166F23A08/send\",\"GEFm8joIAQ8BHQEL1SoCiwDIAAAQuBA=\"]\r, others:map[mqtt_qos:0 mqtt_topic:], timestamp:2023-02-17 11:53:30.25971624 +0000 GMT m=+29.095413264"}
2023-02-17T11:53:31.178Z debug adapter/service.go:162 received a mqtt message {"message": "{data:[1,\"vallydroid/sensors/184166F23A08/send\",\"GEFm8joIAQ8BHQEL1SoCiwDIAAAQuBA=\"]\r, others:map[mqtt_qos:0 mqtt_topic:vallydroid/], timestamp:2023-02-17 11:53:31.178879574 +0000 GMT m=+30.014576617"}
2023-02-17T11:53:31.185Z debug adapter/service.go:162 received a mqtt message {"message": "{data:[1,\"vallydroid/sensors/184166F23A08/send\",\"GEFm8joIAQ8BHQEL1SoCiwDIAAAQuBA=\"]\r, others:map[mqtt_qos:0 mqtt_topic:vallydroid/], timestamp:2023-02-17 11:53:31.185833108 +0000 GMT m=+30.021530151"}
2023-02-17T11:53:32.177Z debug adapter/service.go:148 posting a message to source device {"message": "{data:[1,\"vallydroid/sensors/184166F23A08/send\",\"GEFm8joIAQ8BHQEL1SoCiwDIAAAQuBA=\"]\r, others:map[mqtt_qos:0 mqtt_topic:vallydroid/], timestamp:2023-02-17 11:53:31.178879574 +0000 GMT m=+30.014576617"}
2023-02-17T11:53:32.188Z debug adapter/service.go:148 posting a message to source device {"message": "{data:[1,\"vallydroid/sensors/184166F23A08/send\",\"GEFm8joIAQ8BHQEL1SoCiwDIAAAQuBA=\"]\r, others:map[mqtt_qos:0 mqtt_topic:vallydroid/], timestamp:2023-02-17 11:53:31.185833108 +0000 GMT m=+30.021530151"}
@requeijaum can you share the configuration file?
from this I understand that, you are publishing and subscribing same topic.
- What ever you post on the topic
subscribe
will be sent to the serial device. - Whatever you receive on the serial device, will be published on the topic
publish
- the protocol
raw
simply forwards the data between mqtt and devices, here it is serial.
{"adapterName": "adapter1", "config": {"Name":"","Broker":"tcp://127.0.0.1:1883","InsecureSkipVerify":true,"Username":"","Subscribe":"vallydroid/#","Publish":"vallydroid/","QoS":0,"TransmitPreDelay":"0s","ReconnectDelay":"5s"}}
I think you are looking slimier to serial2mqtt. we have to implement the message parser to support your expectation
@requeijaum I have added format script support. I hope with this option we can achieve your requirement. Still if you are interested in 2mqtt
let me know, I can guide you.
Thanks for the reply!
I need to subscribe: "vallydroid/sensors/+/send" (notice the wildcard)
And post messages for each sub-device at "vallydroid/sensors/MAC_ADDR/recv".
Does 2mqtt have the possibility of publishing messages for different MQTT topics ???
@requeijaum Can you give some of samples of following,
serial
data from your device and how you want to post it tomqtt
(including topic details)mqtt
subscribed data and how you want to post it toserial
device (including mqtt topic details)
Does 2mqtt have the possibility of publishing messages for different MQTT topics ???
It is possible, if you have the topic details on your serial data
First of all:
I receive data from serial port - they are measurements for each sensor.
Since they are base64 coded: I need to decode them internally.
So I publish that data to the MQTT broker (at individual sensor topics).
"vallydroid/sensors/123456789/send" In this example.
After this decode operation: I need to update the sensors screens individually.
I proceed to publish another message in a different sub-topic.
"vallydroid/sensors/123456789/recv", in this case.
I just need to get this message from MQTT to Serial.
That's because all the sensors are running on a different kind of networking and we have a device that acts as a bridge through serial port.
The MQTT broker is a central part of my project because we use it as a way to organize all the sensors involved.
@requeijaum from the log of your initial post I go the following data
-
from serial:
[1,\"vallydroid/sensors/E04066F23A08/send\",\"4EBm8joIAQ0BGgEM1SkCcwC/AAAQrhA=\"]\r
to mqtt:topic: vallydroid/sensors/E04066F23A08/send, data: 4EBm8joIAQ0BGgEM1SkCcwC/AAAQrhA=
-
from mqtt:
topic: vallydroid/sensors/E04066F23A08/recv, data: 4EBm8joIAQ0BGgEM1SkCcwC/AAAQrhA=
to serial:[1,\"vallydroid/sensors/E04066F23A08/recv\",\"4EBm8joIAQ0BGgEM1SkCcwC/AAAQrhA=\"]\r
Am I understand correctly?
That's it. But /recv isn't base64 encoded.
@requeijaum can you try the following configuration with your serial device?
use 2mqtt version 1.4
logger:
mode: development
encoding: console
level: debug
adapters:
- name: my_serial2mqtt
enabled: true
reconnect_delay: 30s
provider: raw
source:
type: serial
port: /dev/ttyUSB0
baud_rate: 115200
transmit_pre_delay: 10ms
mqtt:
broker: tcp://127.0.0.1:1883
insecure: false
username:
password:
subscribe:
publish:
qos: 0
transmit_pre_delay: 0s
reconnect_delay: 5s
formatter_script:
to_mqtt: |
let update_raw_data = raw_data
if (update_raw_data.endsWith("\\r")){
update_raw_data = update_raw_data.slice(0, -2)
}
let serialData = JSON.parse(update_raw_data)
// sample data
// [1,"vallydroid/sensors/E04066F23A08/send","4EBm8joIAQ0BGgEM1SkCcwC/AAAQrhA="]
let keyMap = ["command", "mqtt_topic", "data"]
let serialDataMap = {}
for (let index=0; index<serialData.length;index++) {
serialDataMap[keyMap[index]] = serialData[index]
}
result={
...serialDataMap
}
to_source: |
// generate as follows
// [1,"vallydroid/sensors/E04066F23A08/recv","my-data"]
result={
data: JSON.stringify([1, mqtt_topic, raw_data])
}
I will check this in the next 6 hours. I'm on BRT (Brazilian time).
@requeijaum ok, take your time
I think that the script is running properly !!!
vd@vd1:~/downloads/2mqtt-1.4-linux-arm64$ ./2mqtt
2023-02-24T00:01:28.250Z info start/logger.go:14 welcome to the 2mqtt adapter server :)
2023-02-24T00:01:28.251Z info start/logger.go:15 server information {"version": "{version:1.4, gitCommit:4242702b0b9349e7f457792e3f784ac62a7366a9, buildDate:2023-02-20T02:02:53+00:00, goLang:go1.20.1, platform:linux, arch:arm64}", "logger": {"Mode":"development","Encoding":"console","Level":"debug"}}
2023-02-24T00:01:28.251Z debug adapter/store.go:48 starting an adapter {"name": "adapter1", "provider": "raw"}
2023-02-24T00:01:28.252Z debug mqtt/device.go:60 mqtt config {"adapterName": "adapter1", "config": {"Name":"","Broker":"tcp://127.0.0.1:1883","Insecure":false,"Username":"","Subscribe":"vallydroid/#","Publish":"vallydroid/","QoS":0,"TransmitPreDelay":"0s","ReconnectDelay":"5s"}}
2023-02-24T00:01:28.260Z debug mqtt/device.go:94 mqtt client connected successfully {"adapterName": "adapter1", "timeTaken": "8.139485ms", "clientConfig": {"Name":"","Broker":"tcp://127.0.0.1:1883","Insecure":false,"Username":"","Subscribe":"vallydroid/#","Publish":"vallydroid/","QoS":0,"TransmitPreDelay":"0s","ReconnectDelay":"5s"}}
2023-02-24T00:01:28.261Z info adapter/service.go:238 connected to the mqtt broker {"adapterName": "adapter1", "provider": "raw"}
2023-02-24T00:01:28.261Z debug mqtt/device.go:103 mqtt connection success {"adapterName": "adapter1"}
2023-02-24T00:01:28.261Z debug serial/device.go:58 source device config {"id": "adapter1", "config": {"Port":"/dev/ttyUSB0","BaudRate":115200,"MessageSplitter":10,"TransmitPreDelay":"10ms"}}
2023-02-24T00:01:28.262Z info serial/device.go:62 opening a serial port {"adapterName": "adapter1", "port": "/dev/ttyUSB0"}
2023-02-24T00:01:28.263Z debug mqtt/device.go:186 subscribed a topic {"adapterName": "adapter1", "topic": "vallydroid/#"}
2023-02-24T00:01:28.270Z info adapter/service.go:270 connected to the source device {"adapterName": "adapter1", "provider": "raw"}
2023-02-24T00:01:28.270Z info start/start.go:23 services started {"timeTaken": "18.336869ms"}
^[[21~2023-02-24T00:01:33.857Z debug adapter/service.go:172 received a message from source device {"message": "{data:[1,\"vallydroid/sensors/404266F23A08/send\",\"QEJm8joIABoBKgAU1SUCVv/UAAAQ1xA=\"]\r, others:map[], timestamp:2023-02-24 00:01:33.857024546 +0000 GMT m=+5.619853010", "adapterName": "adapter1", "provider": "raw"}
2023-02-24T00:01:33.860Z debug javascript/utils.go:35 executing script {"variables": {"raw_data":"[1,\"vallydroid/sensors/404266F23A08/send\",\"QEJm8joIABoBKgAU1SUCVv/UAAAQ1xA=\"]\r"}, "scriptString": "let update_raw_data = raw_data\nif (update_raw_data.endsWith(\"\\\\r\")){\n update_raw_data = update_raw_data.slice(0, -2)\n}\nlet serialData = JSON.parse(update_raw_data)\n// sample data\n// [1,\"vallydroid/sensors/E04066F23A08/send\",\"4EBm8joIAQ0BGgEM1SkCcwC/AAAQrhA=\"]\nlet keyMap = [\"command\", \"mqtt_topic\", \"data\"]\nlet serialDataMap = {}\nfor (let index=0; index<serialData.length;index++) {\n serialDataMap[keyMap[index]] = serialData[index]\n}\nresult={\n ...serialDataMap\n}\n"}
2023-02-24T00:01:33.862Z debug javascript/utils.go:50 executed script {"variables": {"raw_data":"[1,\"vallydroid/sensors/404266F23A08/send\",\"QEJm8joIABoBKgAU1SUCVv/UAAAQ1xA=\"]\r"}, "scriptString": "let update_raw_data = raw_data\nif (update_raw_data.endsWith(\"\\\\r\")){\n update_raw_data = update_raw_data.slice(0, -2)\n}\nlet serialData = JSON.parse(update_raw_data)\n// sample data\n// [1,\"vallydroid/sensors/E04066F23A08/send\",\"4EBm8joIAQ0BGgEM1SkCcwC/AAAQrhA=\"]\nlet keyMap = [\"command\", \"mqtt_topic\", \"data\"]\nlet serialDataMap = {}\nfor (let index=0; index<serialData.length;index++) {\n serialDataMap[keyMap[index]] = serialData[index]\n}\nresult={\n ...serialDataMap\n}\n", "output": {"command":1,"data":"QEJm8joIABoBKgAU1SUCVv/UAAAQ1xA=","mqtt_topic":"vallydroid/sensors/404266F23A08/send"}}
2023-02-24T00:01:34.271Z debug mqtt/device.go:134 about to send a message {"adapterName": "adapter1", "message": "{data:QEJm8joIABoBKgAU1SUCVv/UAAAQ1xA=, others:map[command:1 mqtt_qos:0 mqtt_topic:vallydroid/sensors/404266F23A08/send], timestamp:2023-02-24 00:01:33.857024546 +0000 GMT m=+5.619853010"}
2023-02-24T00:01:34.273Z debug adapter/service.go:162 received a mqtt message {"message": "{data:QEJm8joIABoBKgAU1SUCVv/UAAAQ1xA=, others:map[mqtt_qos:0 mqtt_topic:vallydroid//vallydroid/sensors/404266F23A08/send], timestamp:2023-02-24 00:01:34.272683907 +0000 GMT m=+6.035512370", "adapterName": "adapter1", "provider": "raw"}
2023-02-24T00:01:34.275Z debug javascript/utils.go:35 executing script {"variables": {"mqtt_qos":0,"mqtt_topic":"vallydroid//vallydroid/sensors/404266F23A08/send","raw_data":"QEJm8joIABoBKgAU1SUCVv/UAAAQ1xA="}, "scriptString": "// generate as follows\n// [1,\"vallydroid/sensors/E04066F23A08/recv\",\"my-data\"]\nresult={\n data: JSON.stringify([1, mqtt_topic, raw_data])\n}\n"}
2023-02-24T00:01:34.276Z debug javascript/utils.go:50 executed script {"variables": {"mqtt_qos":0,"mqtt_topic":"vallydroid//vallydroid/sensors/404266F23A08/send","raw_data":"QEJm8joIABoBKgAU1SUCVv/UAAAQ1xA="}, "scriptString": "// generate as follows\n// [1,\"vallydroid/sensors/E04066F23A08/recv\",\"my-data\"]\nresult={\n data: JSON.stringify([1, mqtt_topic, raw_data])\n}\n", "output": {"data":"[1,\"vallydroid//vallydroid/sensors/404266F23A08/send\",\"QEJm8joIABoBKgAU1SUCVv/UAAAQ1xA=\"]"}}
2023-02-24T00:01:34.287Z debug adapter/service.go:162 received a mqtt message {"message": "{data:QEJm8joIABoBKgAU1SUCVv/UAAAQ1xA=, others:map[mqtt_qos:0 mqtt_topic:vallydroid//vallydroid/sensors/404266F23A08/send], timestamp:2023-02-24 00:01:34.286979533 +0000 GMT m=+6.049807996", "adapterName": "adapter1", "provider": "raw"}
2023-02-24T00:01:34.289Z debug javascript/utils.go:35 executing script {"variables": {"mqtt_qos":0,"mqtt_topic":"vallydroid//vallydroid/sensors/404266F23A08/send","raw_data":"QEJm8joIABoBKgAU1SUCVv/UAAAQ1xA="}, "scriptString": "// generate as follows\n// [1,\"vallydroid/sensors/E04066F23A08/recv\",\"my-data\"]\nresult={\n data: JSON.stringify([1, mqtt_topic, raw_data])\n}\n"}
2023-02-24T00:01:34.290Z debug javascript/utils.go:50 executed script {"variables": {"mqtt_qos":0,"mqtt_topic":"vallydroid//vallydroid/sensors/404266F23A08/send","raw_data":"QEJm8joIABoBKgAU1SUCVv/UAAAQ1xA="}, "scriptString": "// generate as follows\n// [1,\"vallydroid/sensors/E04066F23A08/recv\",\"my-data\"]\nresult={\n data: JSON.stringify([1, mqtt_topic, raw_data])\n}\n", "output": {"data":"[1,\"vallydroid//vallydroid/sensors/404266F23A08/send\",\"QEJm8joIABoBKgAU1SUCVv/UAAAQ1xA=\"]"}}
2023-02-24T00:01:35.271Z debug adapter/service.go:148 posting a message to source device {"message": "{data:[1,\"vallydroid//vallydroid/sensors/404266F23A08/send\",\"QEJm8joIABoBKgAU1SUCVv/UAAAQ1xA=\"], others:map[], timestamp:2023-02-24 00:01:34.272683907 +0000 GMT m=+6.035512370", "adapterName": "adapter1", "provider": "raw"}
2023-02-24T00:01:35.282Z debug adapter/service.go:148 posting a message to source device {"message": "{data:[1,\"vallydroid//vallydroid/sensors/404266F23A08/send\",\"QEJm8joIABoBKgAU1SUCVv/UAAAQ1xA=\"], others:map[], timestamp:2023-02-24 00:01:34.286979533 +0000 GMT m=+6.049807996", "adapterName": "adapter1", "provider": "raw"}
@requeijaum nice it worked for you. to avoid loopback the same data to 2mqtt
, update the subscribe
topic to vallydroid/sensors/+/recv
Is there a way to subscribe in two topics ?
@requeijaum I see it is possible with comma ,
separated. can you try that?
example:
adapters:
- name: my_serial2mqtt
mqtt:
subscribe: topic1,topic2,topic3
2mqtt/plugin/device/mqtt/device.go
Line 177 in 4242702