LorryStream
About
LorryStream is a light-weight, polyglot event- or stream-processing system, which can be used as a data backplane-, message relay-, and pipeline-subsystem. It is based on Streamz and other modern Python libraries.
You can use LorryStream to stream data received from the network into storage systems, or to relay it to other bus systems. It can be used both as a standalone program, and as a library.
It is conceived in the spirit to generalize the corresponding subsystems of programs and frameworks like Kotori, Wetterdienst, Luftdatenpumpe, amqp-forward, ttnlogger, Kahn, mqttwarn, FIWARE QuantumLeap, or CrateOM.
Motivation
Implement a reusable solution, simple to install and operate, that doesn't depend on vendor-provided infrastructure, and can easily be embedded into existing frameworks and software stacks, or integrated otherwise by running it as a sidecar-service.
Details
- Data sources are message bus systems like AMQP, Kafka, MQTT, ZeroMQ, and network listener endpoints for TCP, HTTP, and WebSocket.
- Data sinks are RDBMS databases supported by SQLAlchemy, or other message brokers.
Caveat
Please note that LorryStream is alpha-quality software, and a work in progress. Contributions of all kinds are very welcome, in order to make it more solid.
Breaking changes should be expected until a 1.0 release, so version pinning is recommended, especially when you use it as a library.
Only a few features sketched out in the README have actually been implemented right now.
Synopsis
The canonical command is lorry relay <source> <sink>
.
Please note %23
is #
.
lorry relay \
"mqtt://localhost/testdrive/%23" \
"crate://localhost/?table=testdrive"
If you prefer a GStreamer-like pipeline definition syntax.
lorry launch "mqttsrc location=mqtt://localhost/testdrive/%23 ! sqlsink location=crate://localhost/?table=testdrive"
Quickstart
If you are in a hurry, and want to run LorryStream without any installation, just use the OCI image on Podman or Docker.
docker run --rm --network=host ghcr.io/daq-tools/lorrystream \
lorry relay \
"mqtt://localhost/testdrive/%23" \
"crate://localhost/?table=testdrive"
Setup
Install lorrystream
from PyPI.
pip install lorrystream
Usage
This section outlines some example invocations of LorryStream, both on the command line, and per library use. Other than the resources available from the web, testing data can be acquired from the repository's testdata folder.
Prerequisites
For properly running some of the example invocations outlined below, you will need a few servers. The easiest way to spin up those instances is to use Podman or Docker.
docker run --name=mosquitto --rm -it --publish=1883:1883 \
eclipse-mosquitto:2.0.15 mosquitto -c /mosquitto-no-auth.conf
-- https://github.com/docker-library/docs/blob/master/eclipse-mosquitto/README.md
docker run --name=cratedb --rm -it --publish=4200:4200 --publish=5432:5432 \
crate:5.2 -Cdiscovery.type=single-node
-- https://github.com/docker-library/docs/blob/master/crate/README.md
Command line use
Help
lorry --help
lorry info
lorry relay --help
Bus to storage
# Relay messages from MQTT to CrateDB.
lorry relay \
"mqtt://localhost/testdrive/%23" \
"crate://localhost/?table=testdrive"
Bus to bus
# Relay messages from AMQP to MQTT.
lorry relay \
"amqp://localhost/testdrive/demo" \
"mqtt://localhost/testdrive/demo"
Library use
>>> from lorrystream import parse_launch
>>> parse_launch("mqttsrc location=mqtt://localhost/testdrive/%23 ! sqlsink location=crate://localhost/?table=testdrive")
OCI
OCI images are available on the GitHub Container Registry (GHCR). We are publishing image variants for general availability- and nightly-releases, and pull requests.
In order to always run the latest nightly
development version, and to use a
shortcut for that, this section outlines how to use an alias for lorry
,
and a variable for storing the data source and sink URIs. It may be useful to
save a few keystrokes on subsequent invocations.
docker pull ghcr.io/daq-tools/lorrystream:nightly
alias lorry="docker run --rm --interactive ghcr.io/daq-tools/lorrystream:nightly lorry"
SOURCE=mqtt://localhost/testdrive/%23
SINK=crate://crate@localhost:4200/?table=testdrive
lorry relay "${SOURCE}" "${SINK}"
Project information
Development
For installing the project from source, please follow the development documentation.
Prior art
There are a many other projects which are aiming at similar goals, and where LorryStream borrows ideas from.