structured-streaming

There are 3 repositories under structured-streaming topic.

• lw-lin / CoolplaySpark

  酷玩 Spark: Spark 源代码解析、Spark 类库等

  sparkspark-streamingstructured-streamingsparkcoreapache-spark
  Language:Scala 3488

• databricks / LearningSparkV2

  This is the github repo for Learning Spark: Lightning-Fast Data Analytics [2nd Edition]

  apache-sparksparkstructured-streamingspark-sqlspark-mllibmllibmlflowdelta-lake
  Language:Scala 1339

• japila-books / spark-structured-streaming-internals

  The Internals of Spark Structured Streaming

  apache-sparkbookinternalsmkdocs-materialsparkstructured-streaming
  417

• Azure / azure-event-hubs-spark

  Enabling Continuous Data Processing with Apache Spark and Azure Event Hubs

  sparkspark-streamingazurescalaeventhubsreal-timestreamingcontinuousapacheapache-sparkmicrosoftevent-hubsconnectordatabricksstreamstructured-streamingbigdataingestionkafka
  Language:Scala 238

• polomarcus / Spark-Structured-Streaming-Examples

  Spark Structured Streaming / Kafka / Cassandra / Elastic

  cassandrakafkasparkspark-sqlstructured-streaming
  Language:Scala 183

• qubole / kinesis-sql

  Kinesis Connector for Structured Streaming

  kinesisreal-time-processingsparkspark-streamingspark-structured-streamingstructured-streaming
  Language:Scala 136

• streamnative / pulsar-spark

  Spark Connector to read and write with Pulsar

  apache-pulsarapache-sparkbatch-processingdata-processingdata-scienceflinksparkspark-sqlstream-processingstructured-streaming
  Language:Scala 113

• chermenin / spark-states

  Custom state store providers for Apache Spark

  apacheapache-sparksparkspark-streamingspark-structured-streamingstatestate-storestatefulstructured-streaming
  Language:Scala 92

• radoslawkrolikowski / financial-market-data-analysis

  Real-Time Financial Market Data Processing and Prediction application

  apache-kafkaapache-sparkfinancial-datamariadbneural-networkspythonpytorchscrapystructured-streaming
  Language:Python 90

• IBM / kafka-streaming-click-analysis

  Use Kafka and Apache Spark streaming to perform click stream analytics

  kafkastructured-streamingapache-sparkibmcodeibm-data-science-experiencedata-scienceclickstreamsparkjupyter-notebook
  Language:Jupyter Notebook 76

• astrolabsoftware / fink-broker

  Astronomy Broker based on Apache Spark

  apache-sparkstructured-streamingapache-kafkaalertsastronomyapache-hbase
  Language:Python 69

• zaleslaw / Spark-Tutorial

  How to build your first Spark application with MLlib, StructuredStreaming, GraphFrames, Datasets and so on? Answer is here!

  kafkasparkstreamingstructured-streaming
  Language:Scala 53

• Klarrio / open-stream-processing-benchmark

  This repository contains the code base for the Open Stream Processing Benchmark.

  stream-processingflinksparkkafka-streamskafkadistributed-systemsdistributed-computingreal-timereal-time-processingstructured-streamingspark-streamingbenchmarkbenchmark-suite
  Language:Jupyter Notebook 50

• HeartSaVioR / spark-sql-kafka-offset-committer

  Kafka offset committer for structured streaming query

  kafkasparkstructured-streaming
  Language:Scala 37

• sankamuk / PysparkCheatsheet

  PySpark Cheatsheet

  apache-sparkpythonstructured-streamingdeltalake
  Language:Python 36

• HeartSaVioR / spark-state-tools

  Spark Structured Streaming State Tools

  apache-sparkstructured-streaming
  Language:Scala 34

• AndrewKuzmin / spark-structured-streaming-examples

  Spark structured streaming examples with using of version 3.5.1

  sparkspark-sqlspark-structured-streamingapache-sparkstructured-streamingdelta-lake
  Language:Scala 26

• aamend / spark-gdelt

  Binding the GDELT universe in a Spark environment

  sparkgdeltparserstructured-streamingnewsanalytics
  Language:Scala 25

• aws-samples / iceberg-streaming-examples

  This repo contains examples of high throughput ingestion using Apache Spark and Apache Iceberg. These examples cover IoT and CDC scenarios using best practices. The code can be deployed into any Spark compatible engine like Amazon EMR Serverless or AWS Glue. A fully local developer environment is also provided.

  apache-icebergapache-sparkstructured-streaming
  Language:Java 23

• mozilla / telemetry-streaming

  Spark Streaming ETL jobs for Mozilla Telemetry

  sparkstreamingrealtime-metricsstructured-streaming
  Language:Scala 18

• sev7e0 / wow-spark

  :high_brightness: spark自学手册，包含了例如spark core、spark sql、spark streaming、spark-kafka、delta-lake，以及scala基础练习，还有一些例如master、shuffle源码分析，总结及翻译。

  sparkspark-sqljavascalastructured-streaming
  Language:Scala 18

• qubole / s3-sqs-connector

  A library for reading data from Amzon S3 with optimised listing using Amazon SQS using Spark SQL Streaming ( or Structured streaming).

  scalasparkstreamingstructured-streamingspark-streamings3sqs
  Language:Scala 17

• qubole / spark-state-store

  Rocksdb state storage implementation for Structured Streaming.

  streamingstructured-streamingsparkstate-managementrocksdbperformancescalabilityqubolereal-time-processing
  Language:Scala 17

• qubole / streaminglens

  Qubole Streaminglens tool for tuning Spark Structured Streaming Pipelines

  cluster-managementmicro-batchesscalaslasparkspark-streamingsparklensstreamingstreaming-pipelinestructured-streaming
  Language:Scala 17

• zekeriyyaa / PySpark-Structured-Streaming-ROS-Kafka-ApacheSpark-Cassandra

  A structured streaming was applied to the robot data from ROS-Gazebo simulation environment using Apache Spark. Data is collected in Kafka, analyzed by Apache Spark and stored in Cassandra.

  apache-sparkapache-kafkaapache-cassandrastructured-streamingkafka-consumerkafka-producerdata-analysispythonpython3pysparkspark-streamingspark-sqlrosros-noeticspark-kafka-integrationspark-cassandraspark-cassandra-connectorspark-kafka-connectorcqlsh
  Language:Python 16

• Neuw84 / structured-streaming-avro-demo

  Spark 3.0.0 Structured Streaming Kafka Avro Demo

  javasparkspark-streamingstructured-streaming
  Language:Java 15

• xiaogp / recsys_structured_streaming

  kafka + structured streaming + phoenix + elasticsearch 基于行为日志实现热门推荐，用户偏好推荐，召回融合策略实现。

  recommender-systemstructured-streaming
  Language:Scala 15

• NashTech-Labs / structured-streaming-application

  Structured Streaming is a reference application showing how to easily integrate structured streaming Apache Spark Structured Streaming, Apache Cassandra and Apache Kafka for fast, structured streaming computations on data.

  scalasparkstructured-streamingkafkacassandradatastax-cassandra-driverknoldus
  Language:Scala 13

• yjshen / spark-connector-test

  A tutorial on how to use pulsar-spark-connector

  apache-pulsarapache-sparkstructured-streamingsparksqlpulsar-spark-connector
  Language:Scala 11

• awslabs / aws-cloudwatch-metrics-custom-spark-listener

  Example Spark streaming sample codes with Custom Listeners to push streaming metrics into Amazon CloudWatch metrics

  streaming-metricscustom-listenersaws-cloudwatchkinesis-streamingkafka-streamsstructured-streaming
  Language:Scala 8

• data-han / twitter_kafka_sentiment

  Ingesting real-time Twitter API using tweepy into Kafka and process using Apache Spark Structured Streaming with Sentiment Analysis TextBlob before loading into time-series database, InfluxDB and monitoring dashboard, Grafana

  grafanainfluxdbkafkareal-timesparkstreamingstructured-streaming
  Language:Python 8

• epishova / Structured-Streaming-Cassandra-Sink

  An example of how to create and use Cassandra sink in Spark Structured Streaming application

  sparkstructured-streamingcassandrasinkscala
  Language:Scala 8

• Rishav273 / kafkaPysparkAnalytics

  Real-time ETL pipeline for financial data (kafka, pyspark) .

  apache-kafkaapache-sparkpysparkstreaming-analyticsstructured-streaming
  Language:Python 8

• chenyyyang / spark-sql-custom-mq-dataSource

  基于Spark 3.1.x 数据源API实现的MQ数据源示例代码

  spark-sqlstructured-streamingdatasourcev2spark3mq
  Language:Java 7

• TrainingByPackt / Big-Data-Processing-with-Apache-Spark-eLearning

  Efficiently tackle large datasets and perform big data analysis with Spark and Python

  sparkpythondatasetspark-mllibrddsstructured-streaming
  Language:Python 7

• thestyleofme / spark-explore

  spark生态学习

  sparkspark-streamingstructured-streaming
  Language:Scala 6