stream-processing

There are 97 repositories under stream-processing topic.

pathway
pathwaycom / pathway
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
batch-processing kafka pathway python streaming machine-learning-algorithms real-time data-analytics data-pipelines data-processing dataflow etl etl-framework iot-analytics rust stream-processing time-series-analysis
Language:Python 43094
google-ai-edge / mediapipe
Cross-platform, customizable ML solutions for live and streaming media.
mediapipe c-plus-plus computer-vision deep-learning android video-processing audio-processing mobile-development machine-learning inference graph-framework graph-based calculator framework pipeline-framework stream-processing perception
Language:C++ 31335
vector
vectordotdev / vector
A high-performance observability data pipeline.
router logs metrics rust observability forwarder vector parser events stream-processing pipeline
Language:Rust 20303
flink-learning
zhisheng17 / flink-learning
flink learning blog. http://www.54tianzhisheng.cn/ 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例，还有 Flink 落地应用的大型项目案例（PVUV、日志存储、百亿数据实时去重、监控告警）分享。欢迎大家支持我的专栏《大数据实时计算引擎 Flink 实战与性能优化》
flink kafka elasticsearch spark redis mysql rocketmq hbase rabbitmq stream-processing streaming clickhouse loki influxdb opentsdb
Language:Java 14929
oxnr / awesome-bigdata
A curated list of awesome big data frameworks, ressources and other awesomeness.
awesome awesome-list bigdata data data-analytics data-science data-stream data-visualization data-warehouse database distributed-database series-database stream-processing streaming-data visualize-data
13871
redpanda
redpanda-data / redpanda
Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
kafka cpp kubernetes go streaming redpanda containers realtime event-driven microservices storage-engine stream-processing seastar
Language:C++ 10996
madd86 / awesome-system-design
A curated list of awesome System Design (A.K.A. Distributed Systems) resources.
relational-database message-broker hadoop-ecosystem nosql interview distributed-systems microservices-architecture stream-processing microservices
10680
watermill
ThreeDotsLabs / watermill
Building event-driven applications the easy way in Go.
cqrs event-driven event-sourcing events go golang kafka nats rabbitmq reactive sagas stream-processing watermill
Language:Go 8878
connect
redpanda-data / connect
Fancy stream processing made operationally mundane
message-queue stream-processing streaming-data message-bus logs stream-processor cqrs event-sourcing go golang kafka amqp rabbitmq nats etl data-engineering data-ops
Language:Go 8447
risingwave
risingwavelabs / risingwave
Real-time event streaming platform. Streaming CDC, stream processing, low-latency serving, and Iceberg management.
database stream-processing rust postgresql kafka materialized-view data-engineering apache-iceberg elt-pipeline etl-pipeline
Language:Rust 8338
fluent-bit
fluent / fluent-bit
Fast and Lightweight Logs, Metrics and Traces processor for Linux, BSD, OSX and Windows
fluentd c logging data-collector fluent-bit forwarder cloudnative stream-processing sql-queries metrics traces logs opentelemetry prometheus
Language:C 7226
robinhood / faust
Python Stream Processing
kafka-streams kafka python asyncio distributed-systems stream-processing
Language:Python 6824
hazelcast / hazelcast
Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.
big-data caching data-in-motion data-insights distributed distributed-computing distributed-systems hacktoberfest hazelcast in-memory java low-latency real-time scalability stream-processing
Language:Java 6391
materialize
MaterializeInc / materialize
Real-time Data Integration and Transformation: use SQL to transform, deliver, and act on fast-changing data.
rust database sql streaming kafka distributed-systems postgresql-dialect materialized-view stream-processing postgresql streaming-data operational-data-store data-store
Language:Rust 6108
apache / hudi
Upserts, Deletes And Incremental Processing on Big Data.
hudi apachehudi datalake bigdata apachespark incremental-processing stream-processing data-integration apacheflink
Language:Java 5936
river
online-ml / river
🌊 Online machine learning in Python
concept-drift data-science incremental-learning machine-learning online-learning online-machine-learning online-statistics python real-time-processing stream-processing streaming streaming-data
Language:Python 5525
fluvio
infinyon / fluvio
🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.
cloud-native streaming rust real-time serverless stateful stream-processing webassembly data-integration data-flow distributed-systems event-driven-architecture stream-processing-engine data-pipelines streaming-data streaming-data-pipelines streaming-data-processing data-analytics streaming-analytics
Language:Rust 5009
danfojs
javascriptdata / danfojs
Danfo.js is an open source, JavaScript library providing high performance, intuitive, and easy to use data structures for manipulating and processing structured data.
data-analytics data-science pandas data-manipulation tensors dataframe javascript data-analysis danfojs stream-data stream-processing tensorflow table plotting-charts
Language:TypeScript 4994
ag2ai / faststream
FastStream is a powerful and easy-to-use Python framework for building asynchronous services interacting with event streams such as Apache Kafka, RabbitMQ, NATS and Redis.
asyncio kafka python rabbitmq asyncapi distributed-systems faststream fastkafka propan nats redis stream-processing
Language:Python 4553
ArroyoSystems / arroyo
Distributed stream processing engine in Rust
data dev-tools infrastructure kafka rust sql stream-processing data-stream-processing stream-processing-engine
Language:Rust 4521
memgraph
memgraph / memgraph
Open-source graph database, tuned for dynamic analytics environments. Easy to adopt, scale and own.
graph-algorithms graph-database graph-analysis stream-processing streaming-data kafka-streams kafka cypher graph nosql opencypher
Language:C++ 3289
manuzhang / awesome-streaming
a curated list of awesome streaming frameworks, applications, etc
awesome awesome-list list stream-processing
2891
douban / dpark
Python clone of Spark, a MapReduce alike framework in Python
bigdata mapreduce dpark stream-processing spark python
Language:Python 2680
peerdb
PeerDB-io / peerdb
Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
etl sql bigquery cloud-native distributed-systems kafka postgres postgresql realtime rust snowflake stream-processing cdc clickhouse s3 eventhubs
Language:Go 2666
pipelinedb / pipelinedb
High-performance time-series aggregation for PostgreSQL
realtime stream-processing analytics sql push postgresql pipelinedb time-series aggregation
Language:C 2648
numaproj / numaflow
Kubernetes-native platform to run massively parallel data/streaming jobs
kubernetes stream-processing data-processing k8s pipeline map-reduce hacktoberfest
Language:Rust 2261
francoispqt / gojay
high performance JSON encoder/decoder with stream API for Golang
go golang decoder encoder perfomance json stream-decoder stream-processing
Language:Go 2131
go-streams
reugn / go-streams
A lightweight stream processing library for Go
aerospike data-pipeline data-stream etl kafka kafka-streams low-code nats-streaming pipeline pulsar redis stream-processing stream-processor streaming-api streaming-data streams throttling websocket windowing workflow
Language:Go 2096
timeplus-io / proton
Fastest SQL pipeline engine in a single C++ binary, for stream processing, analytics, observability and AI.
clickhouse kakfa sql redpanda single-binary stream-processing cpp iceberg etl flink ksqldb observability pipeline real-time
Language:C++ 1895
yomo
yomorun / yomo
🦖 Serverless AI Agent Framework with Geo-distributed Edge AI Infra.
realtime quic stream-processing low-latency serverless distributed-cloud geodistributedsystems yomo function-calling chatgpt openai a2a-protocol claude-code cursor mcp
Language:Go 1839
bytewax / bytewax
Python Stream Processing
python stream-processing rust data-engineering data-processing data-science dataflow machine-learning streaming-data
Language:Python 1812
nerevu / riko
A Python stream processing engine modeled after Yahoo! Pipes
stream-processing etl parallelism rss data asynchronous library cli functional-programming featured
Language:Python 1597
siddhi-io / siddhi
Stream Processing and Complex Event Processing Engine
streaming-data-processing cep stream-processing complex-event-processing online-learning cloud-native cloud library cncf
Language:Java 1562
WallarooLabs / wally
Distributed Stream Processing
stream-processing stream-processor stream-processing-engine pony-language python linux golang go framework api
Language:Pony 1484
quix-streams
quixio / quix-streams
Python Streaming DataFrames for Kafka
data-engineering data-intensive-applications data-science event-driven-architecture kafka machine-learning python real-time-data-processing stream-processing stream-processor streaming-data streaming-data-pipelines streaming-data-processing time-series-data
Language:Python 1441
halaxa / json-machine
Efficient, easy-to-use, and fast PHP JSON stream parser
php stream-processing json-parser json-iterator json-stream parsing
Language:PHP 1242