spark-structured-streaming

There are 5 repositories under spark-structured-streaming topic.

jaceklaskowski / spark-workshop
Apache Spark™ and Scala Workshops
apache-spark spark spark-mllib spark-sql spark-structured-streaming spark-workshops workshop
Language:HTML 253
qubole / kinesis-sql
Kinesis Connector for Structured Streaming
spark structured-streaming kinesis real-time-processing spark-structured-streaming spark-streaming
Language:Scala 137
chermenin / spark-states
Custom state store providers for Apache Spark
spark spark-streaming spark-structured-streaming structured-streaming state-store state stateful apache apache-spark
Language:Scala 92
nama1arpit / reddit-streaming-pipeline
A real-time reddit data streaming pipeline for sentiment analysis of various subreddits
cassandra grafana kafka kubernetes minikube reddit-api spark-structured-streaming terraform
Language:HCL 79
streamnative / awesome-pulsar
A curated list of Pulsar tools, integrations and resources.
apache-pulsar pub-sub messaging apache-bookkeeper apache-kafka apache-flink apache-spark elastic-beats grafana-dashboard prometheus apache-storm spark spark-structured-streaming spark-sql
78
kaantas / spark-twitter-sentiment-analysis
Sentiment Analysis of a Twitter Topic with Spark Structured Streaming
twitter twitter-api twitter-sentiment-analysis apache-spark apache-kafka python pyspark afinn pykafka spark-sql spark-structured-streaming spark kafka twitter-topic sentiment-analysis
Language:Python 54
tomaztk / Azure-Databricks
Azure Databricks - Advent of 2020 Blogposts
python notebooks notebook databricks databricks-notebooks r-language spark sql azure-databricks data-analytics azure-data-factory scala pyspark sparkr spark-structured-streaming data-engineerg machine-learning mllib mlflow azure-machine-learnning
Language:Jupyter Notebook 54
AbsaOSS / hyperdrive
Extensible streaming ingestion pipeline on top of Apache Spark
ingestion spark-structured-streaming spark kafka streaming pipeline framework apache-spark streaming-etl
Language:Scala 41
garystafford / streaming-sales-generator
Streaming Synthetic Sales Data Generator: Streaming sales data generator for Apache Kafka, written in Python
kafka streaming-data python analytics apache-kafka data spark-structured-streaming apache-flink kafka-streams kstreams
Language:Python 38
AndrewKuzmin / spark-structured-streaming-examples
Spark structured streaming examples with using of version 3.5.1
spark spark-sql spark-structured-streaming apache-spark structured-streaming delta-lake
Language:Scala 25
bluejoe2008 / spark-http-stream
spark structured streaming via HTTP communication
spark spark-structured-streaming http
Language:Scala 18
falaybeg / SparkStreaming-Network-Anomaly-Detection
This repository includes supervised and unsupervised machine learning methods which are used to detect anomalies on network datasets. Decision Tree, Random Forest, Gradient Boost Tree, Naive Bayes, and Logistic Regression were used for supervised learning. K-Means was used for unsupervised learning.
anomaly-detection classification clustering decision-tree intrusion-detection machine-learning pyspark spark-streaming spark-structured-streaming supervised-learning unsupervised-learning
Language:Jupyter Notebook 15
LuckyZXL2016 / Spark-Example
Spark1.6和spark2.2的示例，包含kafka,flume,structuredstreaming,jedis,elasticsearch,mysql,dataframe
dataframe elasticsearch jedis kafka mysql spark spark-example spark-sql spark-streaming spark-structured-streaming
Language:Scala 15
rvilla87 / TwitterTrends
Get Twitter trends with twitter4j, stream it to a Kafka topic, save it to MongoDB and visualize in Google Maps
twitter4j trends spark spark-structured-streaming dataframe kafka mongodb pymongo google-maps coordinates ipywidgets scala python
Language:HTML 13
CloudComputingProject-2022 / Data_visualization_and_analysis_tool_for_telemetry_data
An naive anomaly detection and data visualization tool for F1 on board telemetry data.
anomaly-detection fastf1 kafka kafka-consumer kafka-consumers kafka-streams spark spark-sql spark-streaming spark-structured-streaming streamlit streamlit-application streamlit-dashboard streamlit-webapp telemetry
Language:Python 12
spark-structured-streaming-kafka
guidok91 / spark-structured-streaming-kafka
Spark Structured Streaming data pipeline that processes movie ratings data in real-time.
spark etl pyspark real-time data-engineering spark-structured-streaming kafka apache-spark apache-kafka streaming delta-lake
Language:Python 11
stephen29xie / tweet-streaming-data-pipeline
Real-time streaming data pipeline for Twitter Tweets
data-pipeline twitter4j kafka spark-structured-streaming mongodb scala docker
Language:Scala 11
artem0 / kafka-scala-api
Samples for using Kafka within Spark Streaming and Akka Actors, Akka Streams
kafka spark spark-streaming akka-streams scala akka-actors akka message-queue spark-streaming-kafka spark-structured-streaming docker distributed-systems
Language:Scala 9
NashTech-Labs / Sparkathon
A library having Java and Scala examples for Spark 2.x
spark scala java-8 apache-spark knoldus spark-streaming spark-sql spark-dataframes spark-dataset spark-structured-streaming rdd spark-mllib spark-ml
Language:Java 7
AndrewKuzmin / Analytics-For-IoT-Devices-Using-Spark
Analytics for IoT devices using Apache Spark Structured Streaming 2.4.0
spark apache-spark spark-structured-streaming spark-sql
Language:Scala 6
hoseinlook / cpu-anomaly-detection-with-spark
cpu anomaly detection with spark
kafka spark spark-streaming docker pyspark spark-structured-streaming
Language:Python 5
ozancicek / artan
Online latent state estimation with Spark
finite-mixture kalman-filter pyspark python scala spark spark-structured-streaming sparkml streaming
Language:Scala 5
renardeinside / spark-streaming-state-store-example
Spark Structured Streaming with State Store
spark-streaming spark-structured-streaming spark-stream-kafka state-management
Language:Scala 5
AmadeusITGroup / Elastic-Scaling
Elastic scaling is a library that allows to control the number of resources (executors or workers) instantiated by a Spark Structured Streaming Job in order to optimize the effective microbatch duration.
spark spark-structured-streaming
Language:Scala 4
JulienPeloton / mini_spark_broker
Design and proof-of-concept for a Broker for astronomy using Apache Spark
spark spark-structured-streaming kafka python docker
Language:Jupyter Notebook 3
anmollp / Zootopia
A distributed streaming data processing pipeline.
hdfs kafka python3 rest-api spark-structured-streaming
Language:Python 2
dharaneeshvrd / spark-examples
Spark Examples
spark spark-sql spark-example pyspark spark-streaming spark-structured-streaming spark-streaming-kafka
Language:Python 2
fhuertas / uah-mbi-2019-streaming
Repositorio para la clase de UAM, Máster en Business Intelligence, PARALELIZACIÓN DE DATOS, Modulo de Streaming
kafka kafka-stre kafka-connect confluent-platform spark-structured-streaming spark-streaming python kafka-rest
Language:Jupyter Notebook 2
hadiezatpanah / Spark_Java_MostValuableCustomers
This Spark Java project serves as a demonstration of Gradle Spark configuration, specifically focusing on utilizing the MemoryStream class as the streaming source.
gradle java memory-stream spark spark-structured-streaming sparksql
Language:Java 2
haozhang-x / log-analysis-spark
Structured Streaming Log Analysis
spark spark-sql spark-structured-streaming spark-streaming-kafka kafka
Language:Scala 2
iomete / kafka-streaming-job
Kafka streaming job from iomete. This streaming job copies data from Kafka to Iceberg.
iceberg kafka spark-structured-streaming
Language:Python 2
martinKindall / NYC-Taxi-Limousine-Data-Spark
NYC Taxi & Limousine Commission's open data with Spark Streaming 3.0.0
spark spark-streaming kafka spark-kafka nyc-taxi-dataset scala spark-structured-streaming
Language:Scala 2
Pedro-Manoel / iot-analytics-solution-tcc
🎓 Repositório com a solução de IoT Analytics desenvolvida como parte do Trabalho de Conclusão de Curso (TCC) do curso de Ciência da Computação da Universidade Federal de Campina Grande (UFCG)
analytics iot iot-analytics tcc ufcg business-intelligence druid kafka nifi real-time spark spark-structured-streaming superset
Language:TypeScript 2
roksolana-d / spark-streaming-examples
Research on legacy and structured streaming with Spark
scala spark-streaming spark-structured-streaming spark-streaming-kafka kafka
Language:Scala 2
sunujh6 / spark_practice
spark pyspark spark-mllib spark-sql spark-structured-streaming notebook jupyter-notebook
Language:Jupyter Notebook 2
vvittis / CCFD-RF
Credit Card Fraudulent Detection with Random Forest
random-forest-classifier credit-card-fraud machine-learning-algorithms spark-structured-streaming fraud-detection apache-spark apache-spark-framework kafka-streams
Language:Java 2

spark-structured-streaming

jaceklaskowski / spark-workshop

qubole / kinesis-sql

chermenin / spark-states

nama1arpit / reddit-streaming-pipeline

streamnative / awesome-pulsar

kaantas / spark-twitter-sentiment-analysis

tomaztk / Azure-Databricks

AbsaOSS / hyperdrive

garystafford / streaming-sales-generator

AndrewKuzmin / spark-structured-streaming-examples

bluejoe2008 / spark-http-stream

falaybeg / SparkStreaming-Network-Anomaly-Detection

LuckyZXL2016 / Spark-Example

rvilla87 / TwitterTrends

CloudComputingProject-2022 / Data_visualization_and_analysis_tool_for_telemetry_data

guidok91 / spark-structured-streaming-kafka

stephen29xie / tweet-streaming-data-pipeline

artem0 / kafka-scala-api

NashTech-Labs / Sparkathon

AndrewKuzmin / Analytics-For-IoT-Devices-Using-Spark

hoseinlook / cpu-anomaly-detection-with-spark

ozancicek / artan

renardeinside / spark-streaming-state-store-example

AmadeusITGroup / Elastic-Scaling

JulienPeloton / mini_spark_broker

anmollp / Zootopia

dharaneeshvrd / spark-examples

fhuertas / uah-mbi-2019-streaming

hadiezatpanah / Spark_Java_MostValuableCustomers

haozhang-x / log-analysis-spark

iomete / kafka-streaming-job

martinKindall / NYC-Taxi-Limousine-Data-Spark

Pedro-Manoel / iot-analytics-solution-tcc

roksolana-d / spark-streaming-examples

sunujh6 / spark_practice

vvittis / CCFD-RF