RTNiFiStreamProcessors - IoT MQTT sensor stream capture and processing for Apache NiFi
Introduction
These processors can be used to capture and process MQTT streams generated by RTMQTT (https://github.com/richards-tech/RTMQTT) and RTAutomation (https://github.com/richards-tech/RTAutomation). MQTT data is pushed into a flowfile by GetMQTTProcessor and can be used to input any sort of data from MQTT into a NiFi FlowFile. RTMQTTVideoProcessor generates two streams. The first is the actual MJPEG stream as a binary FlowFile, the second is a metadata stream. Normally the sensor and metadata streams would be pushed into Kafka while the video stream would be save to a local file. RTMQTTVideoProcessor generates a timestamp-based filename attribute for use by the PutFile processor. RTMQTTAudioProcessor works in a similar way to RTMQTTVideoProcessor but is intended for a PCM audio over MQTT stream.
ReactToInsteonProcessor
ReactToInsteonProcessor is able to control Instean devices based on the state of other Insteon devices. It communicates back to RTInsteonServer (part of RTAutomation) via the PutMQTTProcessor. The ReactToInsteonProcessor should be fed the status updates from RTInsteonServer. This can be done by using GetMQTTProcessor to capture the stream and connecting the output to ReactToInsteonProcessor.
ReactToInsteonProcessor has a trigger on and trigger off equation. If trigger on becomes true, the controlled device is turned on. If trigger off becomes true, the controlled device is turned off. If both are true, the device is turned on.
The values of the trigger parameters can be simple logic statements. DeviceIDs are the decimal numbers represented by the Insteon IDs if treated as hex numbers without the dots. So, for example, 31.FA.E6 becomes 3275494. Lets say that this is an on/off switch and the idea is to control a device 30.54.19 (3167257). So, in this case:
trigger on: 3275494 * !3167257
trigger off: !3275494 * 3167257
The triggers can be a series of deviceIDs (not limited to two) separated by either "*" (AND) or "+" (OR). "!" can be used as a prefix to the deviceID to invert its state. A level of 0 is regarded as off, 1-255 is on. The controlled device is always set to either 0 by trigger off or 255 by trigger on.
Note that the use of spaces is important. There should be spaces between the deviceIDs and the logical operators but no space between the "!" and the deviceID. Also, AND and OR are given equal precedence. A TODO is to add a proper parser to include operator precedence and parentheses.
NiFi version and patch
This was tested with NiFi 0.4.1. The PutFile processor needs to be patched to support append mode. This repo includes a patch file that can be used for to add the append mode. Copy it to the NiFi source directory and execute this command there:
git apply --stat PutFile.patch
This will report any issues with the patch. If that looks good, the patch can be applied with:
git apply PutFile.patch
This is based on the patch here - https://issues.apache.org/jira/browse/NIFI-958 - but with the file writes implemented differently. When this is applied, the video will cycle through intervals on a basis that can be configured in RTMQTTVideoProcessor and RTMQTTAudioProcessor (second, minute, hour, day).
Build
Clone the repo:
git clone gitL//github.com/richards-tech/RTNiFiStreamProcessors
and build using Maven:
cd RTNiFiStreamProcessors
mvn clean install
This will generate a file ./nifi-richardstech-nar/target/nifi-richardstech-nar-1.0-SNAPSHOT.nar. This should be copied into NiFi's lib directory. The processors will then be available for selection.