spark-streaming

There are 34 repositories under spark-streaming topic.

Angel-ML / angel
A Flexible and Powerful Parameter Server for large-scale machine learning
machine-learning parameter-server spark scala model high-dimensional online-learning spark-streaming
Language:Java 6705
risingwave
risingwavelabs / risingwave
SQL stream processing, analytics, and management. We decouple storage and compute to offer instant failover, dynamic scaling, speedy bootstrapping, and efficient joins.
analytics big-data cloud-native data-engineering database distributed-database etl flink kafka ksqldb materialized-view postgres postgresql real-time real-time-analytics rust serverless spark-streaming sql stream-processing
Language:Rust 6394
lw-lin / CoolplaySpark
酷玩 Spark: Spark 源代码解析、Spark 类库等
spark spark-streaming structured-streaming sparkcore apache-spark
Language:Scala 3446
LuckyZXL2016 / Movie_Recommend
基于Spark的电影推荐系统，包含爬虫项目、web网站、后台管理系统以及spark推荐系统
spark-mllib spark-streaming ssm-maven scrapy scala hadoop nginx hive mysql
Language:Java 2715
dotnet / spark
.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
spark csharp dotnet analytics bigdata spark-streaming spark-sql machine-learning fsharp dotnet-core dotnet-standard streaming apache-spark tpcds tpch azure hdinsight databricks emr microsoft
Language:C# 2001
jacksu / utils4s
scala、spark使用过程中，各种测试用例以及相关资料整理
akka breeze json4s scala scala-demo scala-spark spark spark-streaming
Language:Scala 1085
edp963 / wormhole
Wormhole is a SPaaS (Stream Processing as a Service) Platform
spark-streaming stream-processing wormhole
Language:JavaScript 976
microsoft / Mobius
C# and F# language binding and extensions to Apache Spark
spark apache-spark rdd dataframe dstream dataset streaming csharp mobius kafka-streaming spark-streaming fsharp bigdata mapreduce eventhubs near-real-time
Language:C# 939
cdapio / cdap
An open source framework for building data analytic applications.
unified integration platform dataset mapreduce spark spark-streaming java java-8 cdap python middleware
Language:Java 738
lw-lin / streaming-readings
Streaming System 相关的论文读物
stream-processing streaming flink spark-streaming storm heron dataflow drizzle millwheel s4 apache-spark streaming-engine spe stream-processing-engine
718
Stratio / sparta
Real Time Analytics and Data Pipelines based on Spark Streaming
streaming-data scala stratio spark streaming spark-streaming olap kafka hdfs workflow sparta analytics real-time sparksql stratio-sparta lambda triggers
Language:Scala 524
spirom / LearningSpark
Scala examples for learning to use Spark
scala spark spark-streaming sparkcore sparksql
Language:Scala 440
harbby / sylph
Stream computing platform for bigdata
java sylph flink spark-streaming big-data sql streamsql
Language:Java 403
microsoft / data-accelerator
Data Accelerator for Apache Spark simplifies onboarding to Streaming of Big Data. It offers a rich, easy to use experience to help with creation, editing and management of Spark jobs on Azure HDInsights or Databricks while enabling the full power of the Spark engine.
spark spark-streaming spark-sql sparksql streaming-data streaming servicefabric nodejs docker hdinsight cosmosdb react azure apache-spark iothub eventhub big-data iot kafka kafka-streams
Language:C# 294
databrickslabs / dbldatagen
Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) may be used to generate large simulated / synthetic data sets for test, POCs, and other uses in Databricks environments including in Delta Live Tables pipelines
datagen pyspark python data-generation faker spark spark-streaming delta-live-tables deltalake databricks synthetic-data datageneration datagenerator
Language:Python 268
paypal / gimel
Big Data Processing Framework - Unified Data API or SQL on Any Storage
spark spark-streaming big-data gimel paypal data-api kafka cassandra hbase aerospike elasticsearch jdbc teradata streaming-sql restapi scala pyspark python
Language:Scala 243
Azure / azure-event-hubs-spark
Enabling Continuous Data Processing with Apache Spark and Azure Event Hubs
spark spark-streaming azure scala eventhubs real-time streaming continuous apache apache-spark microsoft event-hubs connector databricks stream structured-streaming bigdata ingestion kafka
Language:Scala 232
mkuthan / example-spark
Spark, Spark Streaming and Spark SQL unit testing strategies
spark spark-streaming testing
Language:Scala 216
Chabane / bigdata-playground
A complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL
docker spark-sql scala kafka hbase parquet avro nodejs angular graphql mongodb machine-learning big-data hadoop apache-spark apache-flink spark-streaming twitter-api python kops
Language:TypeScript 203
spirom / spark-streaming-with-kafka
Self-contained examples of Apache Spark streaming integrated with Apache Kafka.
spark kafka scala spark-streaming
Language:Scala 195
bdp
bluishglc / bdp
A prototype project of big data platform, the source codes of the book Big Data Platform Architecture and Prototype
bigdata prototype quickstart spark spark-streaming spark-sql demo oozie redis kafka spark-demo spark-streaming-examples sqoop sqoop-import sparksql spark-examples middle-end middle-office
Language:Java 190
hortonworks / streamline
StreamLine - Streaming Analytics
streaming real-time storm spark-streaming kafka kafka-streams flink
Language:Java 163
awantik / pyspark-learning
Updated repository
spark spark-streaming pyspark
Language:Jupyter Notebook 157
qubole / kinesis-sql
Kinesis Connector for Structured Streaming
spark structured-streaming kinesis real-time-processing spark-structured-streaming spark-streaming
Language:Scala 137
P7h / Spark-MLlib-Twitter-Sentiment-Analysis
:star2: :sparkles: Analyze and visualize Twitter Sentiment on a world map using Spark MLlib
machine-learning naive-bayes-classification scala spark spark-mllib spark-streaming twitter-sentiment-analysis visualization
Language:Scala 135
mkuthan / example-spark-kafka
Apache Spark and Apache Kafka integration example
spark spark-streaming kafka
Language:Scala 123
LearningJournal / Spark-Streaming-In-Python
Apache Spark 3 - Structured Streaming Course Material
apache-spark spark-streaming spark-sql pyspark python big-data bigdata data-lake
Language:Python 113
wangj1106 / recommendMoteur
电影推荐系统、电影推荐引擎、使用Spark完成的电影推荐引擎
als flume kafka movies recommendation recommendation-engine recommender-system spark spark-sql spark-streaming
Language:Scala 113
snowch / movie-recommender-demo
This project walks through how you can create recommendations using Apache Spark machine learning. There are a number of jupyter notebooks that you can run on IBM Data Science Experience, and there a live demo of a movie recommendation web application you can interact with. The demo also uses IBM Message Hub (kafka) to push application events to topic where they are consumed by a spark streaming job running on IBM BigInsights (hadoop).
cloudant python-flask-application spark spark-streaming hadoop kafka messagehub dsx notebook machine-learning alternating-least-squares bluemix redis collaborative-filtering biginsights ibm-bluemix bokeh hive ibm-biginsights jupyter-notebook
Language:Jupyter Notebook 100
abulbasar / pyspark-examples
Code examples on Apache Spark using python
apache spark spark-streaming python
Language:Jupyter Notebook 98
huangyueranbbc / Spark_ALS
基于spark-ml,spark-mllib,spark-streaming的推荐算法实现
spark spark-mllib als spark-streaming spark-streaming-als
Language:Java 96
chermenin / spark-states
Custom state store providers for Apache Spark
spark spark-streaming spark-structured-streaming structured-streaming state-store state stateful apache apache-spark
Language:Scala 92
huangyueranbbc / SparkDemo
spark全示例代码(java、scala) Spark most full instance code DEMO (java、scala)
bigdata hadoop operator spark spark-sql spark-streaming sparkfun-products sparkjava sparkline sparkp
Language:Java 80
ApacheSpark
martandsingh / ApacheSpark
This repository will help you to learn about databricks concept with the help of examples. It will include all the important topics which we need in our real life experience as a data engineer. We will be using pyspark & sparksql for the development. At the end of the course we also cover few case studies.
apachespark data-analysis data-engineering database databricks datalake deltalake etl etl-pipeline hadoop hive pyspark spark spark-sql spark-streaming sql timetravel
Language:Python 80
jmcmt87 / spark_app_twitter
A data engineering project (Twitter monitor app)
kafka mongodb pyspark s3 spark-streaming altair pandas python
Language:Python 63
garystafford / kafka-connect-msk-demo
For a series of posts on Amazon MSK, Amazon EKS, and Amazon EMR
kafka kafka-connect spark-streaming spark aws kubernetes
Language:Python 62

spark-streaming

Angel-ML / angel

risingwavelabs / risingwave

lw-lin / CoolplaySpark

LuckyZXL2016 / Movie_Recommend

dotnet / spark

jacksu / utils4s

edp963 / wormhole

microsoft / Mobius

cdapio / cdap

lw-lin / streaming-readings

Stratio / sparta

spirom / LearningSpark

harbby / sylph

microsoft / data-accelerator

databrickslabs / dbldatagen

paypal / gimel

Azure / azure-event-hubs-spark

mkuthan / example-spark

Chabane / bigdata-playground

spirom / spark-streaming-with-kafka

bluishglc / bdp

hortonworks / streamline

awantik / pyspark-learning

qubole / kinesis-sql

P7h / Spark-MLlib-Twitter-Sentiment-Analysis

mkuthan / example-spark-kafka

LearningJournal / Spark-Streaming-In-Python

wangj1106 / recommendMoteur

snowch / movie-recommender-demo

abulbasar / pyspark-examples

huangyueranbbc / Spark_ALS

chermenin / spark-states

huangyueranbbc / SparkDemo

martandsingh / ApacheSpark

jmcmt87 / spark_app_twitter

garystafford / kafka-connect-msk-demo