big-data

There are 401 repositories under big-data topic.

awesome-scalability
binhnguyennus / awesome-scalability
The Patterns of Scalable, Reliable, and Performant Large-Scale Systems
architecture awesome awesome-list backend big-data computer-science design-patterns devops distributed-systems interview interview-practice interview-questions lists machine-learning programming resources scalability system system-design web-development
53543
apache / spark
Apache Spark - A unified analytics engine for large-scale data processing
big-data java jdbc python r scala spark sql
Language:Scala 38513
ClickHouse
ClickHouse / ClickHouse
ClickHouse® is a free analytics DBMS for big data
dbms olap analytics sql distributed-database big-data mpp clickhouse hacktoberfest
Language:C++ 34629
donnemartin / data-science-ipython-notebooks
Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.
python machine-learning deep-learning data-science big-data aws tensorflow theano caffe scikit-learn kaggle spark mapreduce hadoop matplotlib pandas numpy scipy keras
Language:Python 26539
apache / flink
Apache Flink
scala java big-data flink python sql
Language:Java 23243
amark / gun
An open source cybersecurity protocol for syncing decentralized graph data.
machine-learning artificial-intelligence big-data blockchain p2p decentralized graph cryptography crypto offline-first realtime crdt protocol database end-to-end encryption dweb dapp web3 metaverse
Language:JavaScript 17822
presto
prestodb / presto
The official home of the Presto distributed SQL query engine for big data
java presto hive hadoop big-data sql data lakehouse query
Language:Java 15622
heibaiying / BigData-Notes
大数据入门指南 :star:
hadoop hdfs yarn mapreduce hive spark storm hbase scala kafka zookeeper flume azkaban sqoop phoenix bigdata big-data
Language:Java 15336
questdb
questdb / questdb
An open source time-series database for fast ingest and SQL queries
time-series low-latency database sql postgres grafana simd questdb tsdb java iot hacktoberfest postgresql cpp analytics time-series-database big-data financial-analysis
Language:Java 13531
andkret / Cookbook
The Data Engineering Cookbook
best-practices big-data cookbook data-engineer data-engineering
12984
apache / predictionio
PredictionIO, a machine learning server for developers and ML engineers.
scala big-data predictionio
Language:Scala 12546
yahoo / CMAK
CMAK is a tool for managing Apache Kafka clusters
big-data cluster-management kafka scala
Language:Scala 11676
nebula
vesoft-inc / nebula
A distributed, fast open-source graph database featuring horizontal scalability and high availability
graph-database distributed database graphdb raft cpp nebula-graph nebula graph nebulagraph big-data distributed-systems scalability hacktoberfest
Language:C++ 10209
trinodb / trino
Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)
java presto hive hadoop big-data sql prestodb database databases distributed-systems distributed-database data-science datalake jdbc query-engine trino analytics delta-lake iceberg
Language:Java 9633
cython / cython
The most widely used Python to C compiler
python cython cpython cpython-extensions c cpp performance big-data
Language:Python 8973
kafka-ui
provectus / kafka-ui
Open-Source Web UI for Apache Kafka Management
kafka-ui kafka-brokers kafka kafka-streams kafka-client opensource kafka-connect kafka-producer streams big-data apache-kafka cluster-management web-ui kafka-manager kafka-cluster streaming-data event-streaming hacktoberfest
Language:Java 8640
StarRocks / starrocks
StarRocks, a Linux Foundation project, is a next-generation sub-second MPP OLAP database for full analytics scenarios, including multi-dimensional analytics, real-time analytics, and ad-hoc queries. InfoWorld’s 2023 BOSSIE Award for best open source software.
database olap sql analytics big-data realtime-database vectorized distributed-database real-time-analytics mpp join star-schema real-time-updates delta-lake hudi iceberg lakehouse datalake lakehouse-platform cloudnative
Language:Java 7973
catboost / catboost
A fast, scalable, high performance Gradient Boosting on Decision Trees library, used for ranking, classification, regression and other machine learning tasks for Python, R, Java, C++. Supports computation on CPU and GPU.
machine-learning decision-trees gradient-boosting gbm gbdt python r kaggle gpu-computing catboost tutorial categorical-features gpu coreml data-science big-data cuda data-mining
Language:Python 7776
apache / beam
Apache Beam is a unified programming model for Batch and Streaming data processing.
batch beam big-data golang java python sql streaming
Language:Java 7565
delta-io / delta
An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
spark acid big-data analytics delta-lake
Language:Scala 6948
h2oai / h2o-3
H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.
h2o machine-learning data-science deep-learning big-data ensemble-learning gbm random-forest naive-bayes pca opensource distributed java python r hadoop spark gpu automl h2o-automl
Language:Jupyter Notebook 6745
risingwave
risingwavelabs / risingwave
SQL stream processing, analytics, and management. We decouple storage and compute to offer speedy bootstrapping, dynamic scaling, time-travel queries, and efficient joins.
database stream-processing cloud-native sql distributed-database rust serverless postgresql real-time postgres flink kafka analytics big-data spark-streaming ksqldb materialized-view data-engineering etl real-time-analytics
Language:Rust 6384
apache / zeppelin
Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.
scala java big-data zeppelin javascript spark flink database nosql
Language:Java 6269
quickwit
quickwit-oss / quickwit
Cloud-native search engine for observability. An open-source alternative to Datadog, Elasticsearch, Loki, and Tempo.
big-data cloud-native cloud-storage distributed-tracing log-management logs open-source rust search-engine tantivy
Language:Rust 6204
arkime / arkime
Arkime is an open source, large scale, full packet capturing, indexing, and database system.
network-monitoring pcap packet-capture nsm javascript c big-data security
Language:JavaScript 6135
pachyderm / pachyderm
Data-Centric Pipelines and Data Versioning
go pachyderm docker analytics big-data containers distributed-systems kubernetes data-science data-analysis
Language:Go 6083
couchdb
apache / couchdb
Seamless multi-master syncing database with an intuitive HTTP/JSON API, designed for reliability
content network-server http cloud erlang javascript couchdb big-data network-client database
Language:Erlang 6041
hazelcast / hazelcast
Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.
java hazelcast in-memory big-data scalability distributed caching hacktoberfest stream-processing low-latency distributed-computing distributed-systems data-in-motion data-insights real-time
Language:Java 5913
vespa
vespa-engine / vespa
AI + Data, online. https://vespa.ai
vespa search-engine big-data ai serving serving-recommendation machine-learning server tensorflow java cpp vector-search
Language:Java 5386
apache / hive
Apache Hive
java hive database sql apache big-data hadoop
Language:Java 5343
feast-dev / feast
The Open Source Feature Store for Machine Learning
machine-learning features ml big-data feature-store python mlops data-engineering data-science data-quality
Language:Python 5293
apache / datafusion
Apache DataFusion SQL Query Engine
arrow big-data dataframe datafusion olap python query-engine rust sql
Language:Rust 5191
SynapseML
microsoft / SynapseML
Simple and Distributed Machine Learning
spark pyspark azure scala microsoft ml machine-learning databricks cognitive-services lightgbm http model-deployment deep-learning ai apache-spark data-science synapse big-data onnx opencv
Language:Scala 4981
tschellenbach / Stream-Framework
Stream Framework is a Python library, which allows you to build news feed, activity streams and notification systems using Cassandra and/or Redis. The authors of Stream-Framework also provide a cloud service for feed technology:
activity-stream cassandra feed news news-feed big-data redis activity-feed
Language:Python 4724
apache / ignite
Apache Ignite
distributed-sql-database iot osgi network-client ignite data-management-platform big-data cloud database network-server hadoop sql cache in-memory-computing in-memory-database
Language:Java 4696
apache / calcite
Apache Calcite
big-data calcite geospatial hadoop java sql
Language:Java 4384