bigdata

There are 300 repositories under bigdata topic.

DataExpert-io / data-engineer-handbook
This is a repo with links to everything you'd ever want to learn about data engineering
apachespark awesome bigdata data dataengineering sql
Language:Jupyter Notebook 38518
TDengine
taosdata / TDengine
High-performance, scalable time-series database designed for Industrial IoT (IIoT) scenarios
iot bigdata time-series database industrial-iot connected-vehicles monitoring tsdb tdengine sql time-series-database scalability cluster metrics financial-analysis cloud-native distributed
Language:C 24459
apache / shardingsphere
Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.
database distributed-database distributed-sql-database sql shard database-cluster mysql postgresql encrypt bigdata data-encryption data-pipeline database-middleware distributed-transaction read-write-splitting database-gateway
Language:Java 20521
heibaiying / BigData-Notes
大数据入门指南 :star:
hadoop hdfs yarn mapreduce hive spark storm hbase scala kafka zookeeper flume azkaban sqoop phoenix bigdata big-data
Language:Java 16734
oxnr / awesome-bigdata
A curated list of awesome big data frameworks, ressources and other awesomeness.
awesome awesome-list bigdata data data-analytics data-science data-stream data-visualization data-warehouse database distributed-database series-database stream-processing streaming-data visualize-data
14012
juicefs
juicedata / juicefs
JuiceFS is a distributed POSIX file system built on top of Redis and S3.
filesystem cloud-native golang redis distributed-systems storage object-storage posix hdfs s3 bigdata go
Language:Go 12375
rustfs / rustfs
🚀 RustFS is an open-source, S3-compatible high-performance object storage system supporting migration and coexistence with other S3-compatible platforms such as MinIO and Ceph.
amazon-s3 bigdata cloud-native filesystem minio object-storage objectstorage rust s3
Language:Rust 11078
wangzhiwubigdata / God-Of-BigData
专注大数据学习面试，大数据成神之路开启。Flink/Spark/Hadoop/Hbase/Hive...
flink spark hadoop hdfs hive hbase kafka zookeeper bigdata flume azkaban
10333
databend
databendlabs / databend
𝗔𝗜-𝗡𝗮𝘁𝗶𝘃𝗲 𝗗𝗮𝘁𝗮 𝗪𝗮𝗿𝗲𝗵𝗼𝘂𝘀𝗲. Blazing analytics, fast search, geo insights, vector AI. Built for multimodal analytics, Open-source Snowflake alternative. https://databend.com
ai bigdata database lakehouse olap rust serverless snowflake sql vector-database
Language:Rust 8971
vaexio / vaex
Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀
dataframe python bigdata tabular-data visualization memory-mapped-file hdf5 machine-learning machinelearning data-science pyarrow
Language:Python 8444
apache / hudi
Upserts, Deletes And Incremental Processing on Big Data.
apacheflink apachehudi apachespark bigdata data-integration datalake hudi incremental-processing stream-processing
Language:Java 6011
volcano-sh / volcano
A Cloud Native Batch System (Project under CNCF)
ai batch-systems bigdata gene golang hpc kubernetes machine-learning serving training
Language:Go 5053
iGaoWei / BigDataView
100+套大数据可视化炫酷大屏Html5模板；包含行业：社区、物业、政务、交通、金融银行等，全网最新、最多，最全、最酷、最炫大数据可视化模板。陆续更新中
bigdata bigdataviewer echarts html-template viewmodel
Language:JavaScript 4634
DTStack / chunjun
A data integration framework
flink bigdata data-integration framework java
Language:Java 4090
liyupi / sql-generator
🔨 用 JSON 来生成结构化的 SQL 语句，基于 Vue3 + TypeScript + Vite + Ant Design + MonacoEditor 实现，项目简单（重逻辑轻页面）、适合练手~
ant-design bigdata hive javascript json monaco-editor mysql spark sql typescript vite vue vue3
Language:Vue 3459
apache / avro
Apache Avro is a data serialization system.
csharp ruby cplusplus python php java c avro bigdata dotnet perl
Language:Java 3175
MoRan1607 / BigDataGuide
大数据学习，从零开始学习大数据，包含大数据学习各阶段学习视频、面试资料
hadoop zookeeper hive flume kafka hbase spark flink scala bigdata javase
3081
douban / dpark
Python clone of Spark, a MapReduce alike framework in Python
bigdata dpark mapreduce python spark stream-processing
Language:Python 2680
griddb / griddb
GridDB is a next-generation open source database that makes time series IoT and big data fast,and easy.
database griddb nosql iot fast time-series newsql sql bigdata timeseries
Language:C++ 2460
dotnet / spark
.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
spark csharp dotnet analytics bigdata spark-streaming spark-sql machine-learning fsharp dotnet-core dotnet-standard streaming apache-spark tpcds tpch azure hdinsight databricks emr microsoft
Language:C# 2085
DTStack / flinkStreamSQL
基于开源的flink，对其实时sql进行扩展；主要实现了流与维表的join，支持原生flink SQL所有的语法
flink sql stream bigdata
Language:Java 2057
shzlw / poli
An easy-to-use BI server built for SQL lovers. Power data analysis in SQL and gain faster business insights.
business-intelligence sql dashboard data-visualization sql-editor jdbc bigdata reporting spring-boot reactjs
Language:Java 1973
byzer-org / byzer-lang
Byzer (former MLSQL): A low-code open-source programming language for data pipeline, analytics and AI.
bigdata machine-learning mlsql sql-like-dsl
Language:Scala 1850
Netflix / genie
Distributed Big Data Orchestration Service
big-data bigdata orchestration configuration configuration-management java spring-boot distributed-systems netflixoss cloud netflix-oss microservice microservices
Language:Java 1758
collabH / bigdata-growth
大数据知识仓库涉及到数据仓库建模、实时计算、大数据、数据中台、系统设计、Java、算法等。
flink kafka hive mapreduce spark olap kudu hadoop hbase debezium hdfs bigdata hudi bigdatalearning
Language:Shell 1688
YoongiKim / AutoCrawler
Google, Naver multiprocess image web crawler (Selenium)
bigdata chromedriver crawler customizable deep-learning google image-crawler multiprocess python selenium thread
Language:Python 1688
jadianes / spark-py-notebooks
Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks
spark python pyspark data-analysis mllib ipython-notebook notebook ipython data-science machine-learning big-data bigdata
Language:Jupyter Notebook 1666
water8394 / BigData-Interview
:dart: :star2:[大数据面试题]分享自己在网络上收集的大数据相关的面试题以及自己的答案总结.目前包含Hadoop/Hive/Spark/Flink/Hbase/Kafka/Zookeeper框架的面试题知识总结
bigdata spark kafka hbase flink hadoop hdfs mapreduce yarn interview interview-questions
1646
apconw / sanic-web
一个轻量级、支持全链路且易于二次开发的大模型应用项目(Large Model Data Assistant) 支持DeepSeek/Qwen3等大模型基于 Dify 、LangChain/LangGraph、Ollama&Vllm、Sanic 和 Text2SQL 📊 等技术构建的一站式大模型应用开发项目，采用 Vue3、TypeScript 和 Vite 5 打造现代UI。它支持通过 ECharts 📈 实现基于大模型的数据图形化问答，具备处理 CSV 文件 📂 表格问答的能力。同时，能方便对接第三方开源 RAG 系统检索系统 🌐等，以支持广泛的通用知识问答。
bigdata dify ollama vllm llm qwen echarts sanic text2sql vue3 python deepseek-r1 mcp lamaindex langchain langgraph-python neo4j
Language:JavaScript 1534
hi-primus / optimus
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
big-data-cleaning bigdata cudf dask dask-cudf data-analysis data-cleaner data-cleaning data-cleansing data-exploration data-extraction data-preparation data-profiling data-science data-transformation data-wrangling machine-learning pyspark spark
Language:Python 1524
tensorbase / tensorbase
TensorBase is a new big data warehousing with modern efforts.
rust bigdata database analytics modern infrastructure data data-infrastructure high-performance engineering rust-lang warehouse data-warehouse
Language:Rust 1458
odd-platform
opendatadiscovery / odd-platform
First open-source data discovery and observability platform. We make a life for data practitioners easy so you can focus on your business.
oss data-platform metadata metadata-management data-pipelines data-engineering observability data-catalog datacatalog data-discovery data-lineage bigdata alerting lineage data-profiling data-exploration data-governance data-quality data-science data-observability
Language:Java 1365
kubernetes-retired / kube-batch
A batch scheduler of kubernetes for high performance workload, e.g. AI/ML, BigData, HPC
kubernetes k8s-sig-scheduling machine-learning bigdata hpc
Language:Go 1092
josonle / Coding-Now
学习记录的一些笔记，以及所看得一些电子书eBooks、视频资源和平常收纳的一些自己认为比较好的博客、网站、工具。涉及大数据几大组件、Python机器学习和数据分析、Linux、操作系统、算法、网络等
ebook-collection notes coding java hadoop-hive spark linux bigdata
Language:Python 1039
apache / celeborn
Apache Celeborn is an elastic and high-performance service for shuffle and spilled data.
bigdata shuffle spark
Language:Java 1013
zeromicro / cds
Data syncing in golang for ClickHouse.
bigdata clickhouse go golang kafka-consumer
Language:Go 977

bigdata

DataExpert-io / data-engineer-handbook

taosdata / TDengine

apache / shardingsphere

heibaiying / BigData-Notes

oxnr / awesome-bigdata

juicedata / juicefs

rustfs / rustfs

wangzhiwubigdata / God-Of-BigData

databendlabs / databend

vaexio / vaex

apache / hudi

volcano-sh / volcano

iGaoWei / BigDataView

DTStack / chunjun

liyupi / sql-generator

apache / avro

MoRan1607 / BigDataGuide

douban / dpark

griddb / griddb

dotnet / spark

DTStack / flinkStreamSQL

shzlw / poli

byzer-org / byzer-lang

Netflix / genie

collabH / bigdata-growth

YoongiKim / AutoCrawler

jadianes / spark-py-notebooks

water8394 / BigData-Interview

apconw / sanic-web

hi-primus / optimus

tensorbase / tensorbase

opendatadiscovery / odd-platform

kubernetes-retired / kube-batch

josonle / Coding-Now

apache / celeborn

zeromicro / cds