Highly configurable data streams in Python 3.x.
Idea is to use JSON schemas for selecting objects from input sources and passing them to action chains.
Both sources and actions implement respective plugin interface, and they can be re-used, configured and glued together in different order via config files.
Priority is to be usable, configurable and human friendly. But according to initial benchmarks this can be also made blazingly fast -- or at least as fast as any custom processing would be.
Overall goal is to battle-tested functional core, easy but powerful configuration language and extendable plugin API modelled after real life data handling scenarios.
These aspects will hopefully make this library worth using.
This is not a message queue. This is a component that sits between message queues, datastores etc.
Also, only basic plugins will be provided here. While usable, they might not adapt to your use-case. Instead of being ready for all possible data formats everywhere, the provided plugins will do just basic (data agnostic) actions. Consider them just a bit more than Hello World examples.
Python 3.x port, asyncio experiments on-going. Stay tuned!
Various stream readers and examples will be provided (HTTP, Redis, SQS, Kombu, local dir, ...)
jsonschema + possibly others like kmatch
Can either data modifiers (modify, sanitize, filter, enrich) or data storers (Redis, disk, databases, message queues, ...)
This project is a full rewrite of an earlier, abandoned Python 2.x project.
Pieces will be committed here once things get ported into Python 3.6+ syntax, and utilizing latest and greatest helper libraries.
Once the basic pieces exist, it is possible to extend the flow by having statistics, throttling, logging, etc plugins too.