stream-processing

There are 82 repositories under stream-processing topic.

google-ai-edge / mediapipe
Cross-platform, customizable ML solutions for live and streaming media.
android audio-processing c-plus-plus calculator computer-vision deep-learning framework graph-based graph-framework inference machine-learning mediapipe mobile-development perception pipeline-framework stream-processing video-processing
Language:C++ 25732
vector
vectordotdev / vector
A high-performance observability data pipeline.
events forwarder logs metrics observability parser pipeline router rust stream-processing vector
Language:Rust 16705
flink-learning
zhisheng17 / flink-learning
flink learning blog. http://www.54tianzhisheng.cn/ 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例，还有 Flink 落地应用的大型项目案例（PVUV、日志存储、百亿数据实时去重、监控告警）分享。欢迎大家支持我的专栏《大数据实时计算引擎 Flink 实战与性能优化》
clickhouse elasticsearch flink hbase influxdb kafka loki mysql opentsdb rabbitmq redis rocketmq spark stream-processing streaming
Language:Java 14296
oxnr / awesome-bigdata
A curated list of awesome big data frameworks, ressources and other awesomeness.
awesome bigdata data-warehouse data-analytics data-visualization data-stream database data-science data awesome-list distributed-database series-database stream-processing streaming-data visualize-data
12854
redpanda
redpanda-data / redpanda
Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
containers cpp event-driven go kafka kubernetes microservices realtime redpanda seastar storage-engine stream-processing streaming
Language:C++ 8960
madd86 / awesome-system-design
A curated list of awesome System Design (A.K.A. Distributed Systems) resources.
distributed-systems hadoop-ecosystem interview message-broker microservices microservices-architecture nosql relational-database stream-processing
8560
benthos
benthosdev / benthos
Fancy stream processing made operationally mundane
amqp cqrs data-engineering data-ops etl event-sourcing go golang kafka logs message-bus message-queue nats rabbitmq stream-processing stream-processor streaming-data
Language:Go 7710
watermill
ThreeDotsLabs / watermill
Building event-driven applications the easy way in Go.
cqrs event-driven event-sourcing events go golang kafka nats rabbitmq reactive sagas stream-processing watermill
Language:Go 6807
robinhood / faust
Python Stream Processing
kafka-streams kafka python asyncio distributed-systems stream-processing
Language:Python 6680
risingwave
risingwavelabs / risingwave
SQL stream processing, analytics, and management. We decouple storage and compute to offer instant failover, dynamic scaling, speedy bootstrapping, and efficient joins.
analytics big-data cloud-native data-engineering database distributed-database etl flink kafka ksqldb materialized-view postgres postgresql real-time real-time-analytics rust serverless spark-streaming sql stream-processing
Language:Rust 6401
hazelcast / hazelcast
Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.
big-data caching data-in-motion data-insights distributed distributed-computing distributed-systems hacktoberfest hazelcast in-memory java low-latency real-time scalability stream-processing
Language:Java 5928
confluentinc / ksql
The database purpose-built for stream processing applications.
event-streaming-database interactive kafka kafka-connect ksqldb ksqldb-documentation ksqldb-tutorials materialized-views real-time sql stream-processing streaming-queries
Language:Java 5831
materialize
MaterializeInc / materialize
The data warehouse for operational workloads.
rust database sql streaming kafka distributed-systems postgresql-dialect materialized-view stream-processing postgresql operational-data-warehouse data-warehouse streaming-data
Language:Rust 5608
fluent / fluent-bit
Fast and Lightweight Logs and Metrics processor for Linux, BSD, OSX and Windows
fluentd c logging data-collector fluent-bit forwarder cloudnative stream-processing sql-queries metrics traces logs opentelemetry prometheus
Language:C 5413
apache / hudi
Upserts, Deletes And Incremental Processing on Big Data.
apacheflink apachehudi apachespark bigdata data-integration datalake hudi incremental-processing stream-processing
Language:Java 5119
river
online-ml / river
🌊 Online machine learning in Python
concept-drift data-science incremental-learning machine-learning online-learning online-machine-learning online-statistics python real-time-processing stream-processing streaming streaming-data
Language:Python 4813
danfojs
javascriptdata / danfojs
Danfo.js is an open source, JavaScript library providing high performance, intuitive, and easy to use data structures for manipulating and processing structured data.
data-analytics data-science pandas data-manipulation tensors dataframe javascript data-analysis danfojs stream-data stream-processing tensorflow table plotting-charts
Language:TypeScript 4674
ArroyoSystems / arroyo
Distributed stream processing engine in Rust
data data-stream-processing dev-tools infrastructure kafka rust sql stream-processing stream-processing-engine
Language:Rust 3344
fluvio
infinyon / fluvio
Lean and mean distributed stream processing system written in rust and web assembly.
cloud-native data-flow data-integration data-pipelines distributed-systems event-driven-architecture real-time rust serverless stateful stream-processing stream-processing-engine streaming streaming-data streaming-data-pipelines streaming-data-processing webassembly
Language:Rust 2695
douban / dpark
Python clone of Spark, a MapReduce alike framework in Python
bigdata dpark mapreduce python spark stream-processing
Language:Python 2693
pipelinedb / pipelinedb
High-performance time-series aggregation for PostgreSQL
aggregation analytics pipelinedb postgresql push realtime sql stream-processing time-series
Language:C 2616
manuzhang / awesome-streaming
a curated list of awesome streaming frameworks, applications, etc
awesome awesome-list list stream-processing
2572
memgraph
memgraph / memgraph
Open-source graph database, tuned for dynamic analytics environments. Easy to adopt, scale and own.
cypher graph graph-algorithms graph-analysis graph-database kafka kafka-streams nosql opencypher stream-processing streaming-data
Language:C++ 2136
francoispqt / gojay
high performance JSON encoder/decoder with stream API for Golang
decoder encoder go golang json perfomance stream-decoder stream-processing
Language:Go 2106
airtai / faststream
FastStream is a powerful and easy-to-use Python framework for building asynchronous services interacting with event streams such as Apache Kafka, RabbitMQ, NATS and Redis.
asyncapi asyncio distributed-systems fastkafka faststream kafka nats propan python rabbitmq redis stream-processing
Language:Python 1863
peerdb
PeerDB-io / peerdb
Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
bigquery cdc clickhouse cloud-native distributed-systems etl eventhubs kafka postgres postgresql realtime rust s3 snowflake sql stream-processing
Language:Go 1849
go-streams
reugn / go-streams
A lightweight stream processing library for Go
aerospike data-pipeline data-stream etl kafka kafka-streams low-code nats-streaming pipeline pulsar redis stream-processing stream-processor streaming-api streaming-data streams throttling websocket windowing workflow
Language:Go 1764
yomo
yomorun / yomo
🦖 Stateful Serverless Framework for building Geo-distributed Edge AI Infra
edge-computing realtime quic stream-processing low-latency serverless distributed-cloud geodistributedsystems webassembly yomo function-calling chatgpt gemini gpt-4o openai
Language:Go 1622
nerevu / riko
A Python stream processing engine modeled after Yahoo! Pipes
asynchronous cli data etl featured functional-programming library parallelism rss stream-processing
Language:Python 1608
siddhi-io / siddhi
Stream Processing and Complex Event Processing Engine
streaming-data-processing cep stream-processing complex-event-processing online-learning cloud-native cloud library cncf
Language:Java 1500
WallarooLabs / wally
Distributed Stream Processing
api framework go golang linux pony-language python stream-processing stream-processing-engine stream-processor
Language:Pony 1479
timeplus-io / proton
A streaming SQL engine, a fast and lightweight alternative to ksqlDB and Apache Flink, 🚀 powered by ClickHouse.
analytics clickhouse confluent cpp flink-alternative high-performance kakfa ksqldb-alternative redpanda simd single-binary sql stream-processing streaming-sql udf
Language:C++ 1334
bytewax / bytewax
Python Stream Processing
data-engineering data-processing data-science dataflow machine-learning python rust stream-processing streaming-data
Language:Python 1259
hazelcast-jet
hazelcast / hazelcast-jet
Distributed Stream and Batch Processing
java big-data stream-processing batch-processing event-processing low-latency kafka cdc hacktoberfest
Language:Java 1091
spring-cloud / spring-cloud-dataflow
A microservices-based Streaming and Batch data processing in Cloud Foundry and Kubernetes
microservices-architecture orchestration stream-processing batch-processing predictive-analytics cloud-native datapipelines
Language:Java 1078
halaxa / json-machine
Efficient, easy-to-use, and fast PHP JSON stream parser
php stream-processing json-parser json-iterator json-stream parsing
Language:PHP 1026