stream-processing

There are 81 repositories under stream-processing topic.

google / mediapipe
Cross-platform, customizable ML solutions for live and streaming media.
mediapipe c-plus-plus computer-vision deep-learning android video-processing audio-processing mobile-development machine-learning inference graph-framework graph-based calculator framework pipeline-framework stream-processing perception
Language:C++ 25396
vector
vectordotdev / vector
A high-performance observability data pipeline.
router logs metrics rust observability forwarder vector parser events stream-processing pipeline
Language:Rust 16426
flink-learning
zhisheng17 / flink-learning
flink learning blog. http://www.54tianzhisheng.cn/ 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例，还有 Flink 落地应用的大型项目案例（PVUV、日志存储、百亿数据实时去重、监控告警）分享。欢迎大家支持我的专栏《大数据实时计算引擎 Flink 实战与性能优化》
flink kafka elasticsearch spark redis mysql rocketmq hbase rabbitmq stream-processing streaming clickhouse loki influxdb opentsdb
Language:Java 14247
newTendermint / awesome-bigdata
A curated list of awesome big data frameworks, ressources and other awesomeness.
awesome bigdata data-warehouse data-analytics data-visualization data-stream database data-science data awesome-list distributed-database series-database stream-processing streaming-data visualize-data
12786
redpanda
redpanda-data / redpanda
Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
kafka cpp kubernetes go streaming redpanda containers realtime event-driven microservices storage-engine stream-processing seastar
Language:C++ 8769
madd86 / awesome-system-design
A curated list of awesome System Design (A.K.A. Distributed Systems) resources.
relational-database message-broker hadoop-ecosystem nosql interview distributed-systems microservices-architecture stream-processing microservices
8271
benthos
benthosdev / benthos
Fancy stream processing made operationally mundane
message-queue stream-processing streaming-data message-bus logs stream-processor cqrs event-sourcing go golang kafka amqp rabbitmq nats etl data-engineering data-ops
Language:Go 7552
watermill
ThreeDotsLabs / watermill
Building event-driven applications the easy way in Go.
watermill event-driven event-sourcing sagas events stream-processing reactive go golang rabbitmq kafka cqrs nats
Language:Go 6709
robinhood / faust
Python Stream Processing
kafka-streams kafka python asyncio distributed-systems stream-processing
Language:Python 6671
risingwave
risingwavelabs / risingwave
Cloud-native SQL stream processing, analytics, and management. KsqlDB and Apache Flink alternative. 🚀 10x more productive. 🚀 10x more cost-efficient.
database stream-processing cloud-native sql distributed-database rust serverless postgresql real-time postgres postgresql-database flink kafka analytics big-data spark-streaming ksqldb spark materialized-view data-engineering
Language:Rust 6280
hazelcast / hazelcast
Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.
java hazelcast in-memory big-data scalability distributed caching hacktoberfest stream-processing low-latency distributed-computing distributed-systems data-in-motion data-insights real-time
Language:Java 5860
confluentinc / ksql
The database purpose-built for stream processing applications.
stream-processing real-time sql interactive kafka kafka-connect materialized-views ksqldb ksqldb-documentation ksqldb-tutorials streaming-queries event-streaming-database
Language:Java 5811
materialize
MaterializeInc / materialize
The data warehouse for operational workloads.
rust database sql streaming kafka distributed-systems postgresql-dialect materialized-view stream-processing postgresql operational-data-warehouse data-warehouse streaming-data
Language:Rust 5562
fluent / fluent-bit
Fast and Lightweight Logs and Metrics processor for Linux, BSD, OSX and Windows
fluentd c logging data-collector fluent-bit forwarder cloudnative stream-processing sql-queries metrics traces logs opentelemetry prometheus
Language:C 5321
apache / hudi
Upserts, Deletes And Incremental Processing on Big Data.
hudi apachehudi datalake bigdata apachespark incremental-processing stream-processing data-integration apacheflink
Language:Java 5053
river
online-ml / river
🌊 Online machine learning in Python
incremental-learning machine-learning python online-learning online-statistics data-science streaming online-machine-learning streaming-data concept-drift real-time-processing stream-processing
Language:Python 4761
danfojs
javascriptdata / danfojs
Danfo.js is an open source, JavaScript library providing high performance, intuitive, and easy to use data structures for manipulating and processing structured data.
data-analytics data-science pandas data-manipulation tensors dataframe javascript data-analysis danfojs stream-data stream-processing tensorflow table plotting-charts
Language:TypeScript 4645
ArroyoSystems / arroyo
Distributed stream processing engine in Rust
data dev-tools infrastructure kafka rust sql stream-processing data-stream-processing stream-processing-engine
Language:Rust 3261
douban / dpark
Python clone of Spark, a MapReduce alike framework in Python
bigdata dpark mapreduce python spark stream-processing
Language:Python 2693
fluvio
infinyon / fluvio
Lean and mean distributed stream processing system written in rust and web assembly.
cloud-native data-flow data-integration data-pipelines distributed-systems event-driven-architecture real-time rust serverless stateful stream-processing stream-processing-engine streaming streaming-data streaming-data-pipelines streaming-data-processing webassembly
Language:Rust 2624
pipelinedb / pipelinedb
High-performance time-series aggregation for PostgreSQL
realtime stream-processing analytics sql push postgresql pipelinedb time-series aggregation
Language:C 2614
manuzhang / awesome-streaming
a curated list of awesome streaming frameworks, applications, etc
awesome awesome-list list stream-processing
2556
francoispqt / gojay
high performance JSON encoder/decoder with stream API for Golang
decoder encoder go golang json perfomance stream-decoder stream-processing
Language:Go 2104
memgraph
memgraph / memgraph
Open-source graph database, built for real-time streaming data, compatible with Neo4j.
graph-algorithms graph-database graph-analysis stream-processing streaming-data kafka-streams kafka cypher graph nosql opencypher
Language:C++ 2076
airtai / faststream
FastStream is a powerful and easy-to-use Python framework for building asynchronous services interacting with event streams such as Apache Kafka, RabbitMQ, NATS and Redis.
asyncapi asyncio distributed-systems fastkafka faststream kafka nats propan python rabbitmq redis stream-processing
Language:Python 1772
go-streams
reugn / go-streams
A lightweight stream processing library for Go
stream-processing pipeline etl kafka data-stream kafka-streams streams redis pulsar data-pipeline streaming-data stream-processor aerospike websocket nats-streaming streaming-api windowing throttling low-code workflow
Language:Go 1746
nerevu / riko
A Python stream processing engine modeled after Yahoo! Pipes
stream-processing etl parallelism rss data asynchronous library cli functional-programming featured
Language:Python 1609
yomo
yomorun / yomo
🦖 Stateful Serverless Framework for building Geo-distributed Edge AI Infra
edge-computing realtime quic stream-processing low-latency serverless distributed-cloud geodistributedsystems edge-ai geo-distributed webassembly yomo function-calling llm-inference
Language:Go 1609
peerdb
PeerDB-io / peerdb
Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
bigquery cdc clickhouse cloud-native distributed-systems etl eventhubs kafka postgres postgresql realtime rust s3 snowflake sql stream-processing
Language:Go 1598
siddhi-io / siddhi
Stream Processing and Complex Event Processing Engine
cep cloud cloud-native cncf complex-event-processing library online-learning stream-processing streaming-data-processing
Language:Java 1493
WallarooLabs / wally
Distributed Stream Processing
stream-processing stream-processor stream-processing-engine pony-language python linux golang go framework api
Language:Pony 1481
timeplus-io / proton
A streaming SQL engine, a fast and lightweight alternative to ksqlDB and Apache Flink, 🚀 powered by ClickHouse.
analytics clickhouse confluent cpp flink-alternative high-performance kakfa ksqldb-alternative redpanda simd single-binary sql stream-processing streaming-sql udf
Language:C++ 1270
bytewax / bytewax
Python Stream Processing
python stream-processing rust data-engineering data-processing data-science dataflow machine-learning streaming-data
Language:Python 1137
hazelcast-jet
hazelcast / hazelcast-jet
Distributed Stream and Batch Processing
java big-data stream-processing batch-processing event-processing low-latency kafka cdc hacktoberfest
Language:Java 1089
spring-cloud / spring-cloud-dataflow
A microservices-based Streaming and Batch data processing in Cloud Foundry and Kubernetes
microservices-architecture orchestration stream-processing batch-processing predictive-analytics cloud-native datapipelines
Language:Java 1066
clj-commons / manifold
A compatibility layer for event-driven abstractions
async clojure future hacktoberfest stream stream-processing
Language:Clojure 1007