datalake

There are 23 repositories under datalake topic.

Sinaptik-AI / pandas-ai
Chat with your database (SQL, CSV, pandas, polars, mongodb, noSQL, etc). PandasAI makes data analysis conversational using LLMs (GPT 3.5 / 4, Anthropic, VertexAI) and RAG.
llm pandas ai data-analysis data-science gpt-3 gpt-4 csv data sql database datalake
Language:Python 10962
trinodb / trino
Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)
analytics big-data data-science database databases datalake delta-lake distributed-database distributed-systems hadoop hive iceberg java jdbc presto prestodb query-engine sql trino
Language:Java 9561
StarRocks / starrocks
StarRocks, a Linux Foundation project, is a next-generation sub-second MPP OLAP database for full analytics scenarios, including multi-dimensional analytics, real-time analytics, and ad-hoc queries. InfoWorld’s 2023 BOSSIE Award for best open source software.
analytics big-data cloudnative database datalake delta-lake distributed-database hudi iceberg join lakehouse lakehouse-platform mpp olap real-time-analytics real-time-updates realtime-database sql star-schema vectorized
Language:Java 7760
activeloopai / deeplake
Database for AI. Store Vectors, Images, Texts, Videos, etc. Use with LLMs/LangChain. Store, query, version, & visualize any AI data. Stream data in real-time to PyTorch/TensorFlow. https://activeloop.ai
ai computer-vision cv data-science data-version-control datalake datasets deep-learning image-processing langchain large-language-models llm machine-learning ml mlops python pytorch tensorflow vector-database vector-search
Language:Python 7704
apache / hudi
Upserts, Deletes And Incremental Processing on Big Data.
hudi apachehudi datalake bigdata apachespark incremental-processing stream-processing data-integration apacheflink
Language:Java 5064
lakeFS
treeverse / lakeFS
lakeFS - Data version control for your data lake | Git for data
data-engineering data-versioning go object-storage data-lake aws-s3 data-quality azure-blob-storage google-cloud-storage golang git-for-data lakefs apache-spark apache-sparksql hadoop-filesystem datalake datalakes data-version-control azure-storage
Language:Go 4064
paradedb / paradedb
Postgres for Search and Analytics
aggregations analytics big-data bm25 database datalake elasticsearch faceting full-text-search htap hybrid-search object-storage olap postgres real-time-analytics self-hosted similarity-search sparse-vector splade sql
Language:Rust 3837
dinky
DataLinkDC / dinky
Dinky is a real-time data development platform based on Apache Flink, enabling agile data development, deployment and operation.
datalake datawarehouse flink flinkcdc flinksql olap real-time-computing-platform sql
Language:Java 2811
lakesoul-io / LakeSoul
LakeSoul is an end-to-end, realtime and cloud native Lakehouse framework with fast data ingestion, concurrent update and incremental data analytics on cloud storages for both BI and AI applications.
lakesoul datalake lakehouse spark flink streaming big-data postgresql rust sql huggingface python pytorch arrow datafusion vectorized velox
Language:Java 2305
leo-project / leofs
The LeoFS Storage System
leofs erlang distributed-storage distributed-file-system s3-storage s3 nfs-server nfs datalake
Language:Erlang 1538
zingg
zinggAI / zingg
Scalable identity resolution, entity resolution, data mastering and deduplication using ML
fuzzymatch fuzzy-matching deduplication dedupe masterdata dataengineering data-transformation analytics-engineering entity-resolution identity-resolution data-transformations data-science spark ml etl dataquality identity modern-data-stack analytics datalake
Language:Java 877
apache / amoro
Apache Amoro (incubating) is a Lakehouse management system built on open data lake formats.
bigdata datalake lakehouse
Language:Java 690
leesf / hudi-resources
汇总Apache Hudi相关资料
hudi apachehudi apache hudi-resources datalake bigdata stream-processing incremental-processing data-integration
518
automate-dv
Datavault-UK / automate-dv
A free to use dbt package for creating and loading Data Vault 2.0 compliant Data Warehouses (powered by dbt, an open source data engineering tool, registered trademark of dbt Labs)
datavault20 dbt datawarehousing datavault etl snowflake dataengineering datawarehouse datalake elt data-vault sql metadata
456
datastrato / gravitino
World's most powerful data catalog service with providing a high-performance, geo-distributed and federated metadata lake.
datalake lakehouse metadata federated-query stratosphere metalake skycomputing data-catalog ai-catalog model-catalog
Language:Java 316
cuebook / cuelake
Use SQL to build ELT pipelines on a data lakehouse.
apache-iceberg delta lakehouse datalake data-lake elt etl data-engineering data-integration data-ingestion apache-spark spark-sql upsert incremental-updates data-transfer pipelines data-pipeline zeppelin-notebook sql
Language:JavaScript 283
linkedin / openhouse
Open Control Plane for Tables in Data Lakehouse
big-data catalog datalake datalakehouse declarative iceberg management tables
Language:Java 244
japila-books / delta-lake-internals
The Internals of Delta Lake
deltalake book internals delta-lake books datalake
175
awslabs / aws-orbit-workbench
A Data Platform built for AWS, powered by Kubernetes.
kubernetes eks jupyter jupyterhub data-analysis datalake mach orbit-workbench gpu eks-cluster aws dataengineering redshift analytics workbench
Language:Python 127
UncoderIO / Uncoder_IO
An IDE and translation engine for detection engineers and threat hunters. Be faster, write smarter, keep 100% privacy.
datalake edr roota siem sigma threathunting translation xdr uncoder uncoderio
Language:Python 104
UncoderIO / RootA
Roota is a public-domain language of threat detection and response that combines native queries from a SIEM, EDR, XDR, or Data Lake with standardized metadata and threat intelligence to enable automated translation into other languages
datalake edr roota rootalanguage siem xdr
101
Real-time-Data-Warehouse
izhangzhihao / Real-time-Data-Warehouse
Real-time Data Warehouse with Apache Flink & Apache Kafka & Apache Hudi
flink data-warehouse real-time-data-warehouse data-warehousing flink-sql debezium kafka elasticsearch delta-lake cdc change-data-capture hudi hoodie iceberg sql datalake delta deltalake spark spark-sql
Language:Dockerfile 99
WeBankFinTech / Streamis
Streaming application development and management system, based on Linkis and DSS, planning to provide the workflow-like graphical drag-and-drop development capability.
flink linkis dataspherestudio wedatasphere streamis streaming hudi iceberg datalake warehouse kafka deltalake
Language:Java 97
LearningJournal / SparkProgrammingInScala
Apache Spark Course Material
apache-spark spark spark-sql spark-scala big-data bigdata datalake data-lake scala
Language:Scala 81
ApacheSpark
martandsingh / ApacheSpark
This repository will help you to learn about databricks concept with the help of examples. It will include all the important topics which we need in our real life experience as a data engineer. We will be using pyspark & sparksql for the development. At the end of the course we also cover few case studies.
apachespark data-analysis data-engineering database databricks datalake deltalake etl-pipeline hadoop hive spark spark-sql spark-streaming timetravel etl pyspark sql
Language:Python 81
apache / doris-website
Apache Doris Website
doris analytics apache big-data data-warehousing database datalake dbms distributed-system hadoop hive hudi iceberg mpp olap ssb tpch vectorized
Language:TypeScript 64
vim89 / datapipelines-essentials-python
Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations
apache-spark big-data data-pipeline datalake etl etl-components etl-framework etl-pipeline hadoop hadoop-hdfs hadoop-mapreduce pyspark python python3 spark spark-sql xml xml-parsing
Language:Python 53
pracdata / awesome-open-source-data-engineering
A curated list of open source tools used in analytical stacks and data engineering ecosystem
analytics awesome awesome-list data data-analytics data-engineering data-integration data-lakehouse data-pipeline data-platform data-storage data-warehouse database datalake etl lakehouse mlops observability self-hosted workflow-engine
51
fuslab / anyscale
anyscale roadmap
fusiondb spark rdbms datalake
49
hifxit / dataligo
A library to accelerate ML and ETL pipeline by connecting all data sources
database datalake datawarehouse etl-pipeline ml-pipeline nosql python
Language:Python 47
GitDataAI / jiaozifs
An Git-like version control file system for data lineage & data collaboration.
data-collaboration data-versioning aiops data-mesh data-product dataops digital-twins enterprise-datahub mlops federated-learning data-lake-management data-version-control data-lake datalake git-for-data git-filesystem jiaozifs version-controlled-filesystem data-lineage
Language:Go 42
PaloAltoNetworks / pan-cortex-data-lake-python
Python idiomatic SDK for Cortex™ Data Lake.
pancloud paloaltonetworks applicationframework python sdk api rest-api panw pan logging-service event-service directory-sync-service directory-sync paloalto logging directory event cortex datalake data
Language:Python 41
LearningJournal / Spark-Streaming-In-Scala
Apache Spark 3 - Structured Streaming Course Material
apache-spark spark-streaming spark-sql scala spark big-data bigdata datalake
Language:Scala 38
rlevchenko / terraform-azure-data
Terraform script to deploy almost all Azure Data Services
servicebus datalake datafactory eventhub databricks azurefunctions azuredataexplorer kusto azureanalysisserver eventgrid azure-data-factory azure-resources data-lake-storage azure lrs
Language:HCL 36
DataTech-Solutions / Threat-Detection-and-Visualization
Threat Detection and Visualization
api datafactory datalake defender deltalake keyvault parquet-files postman powerbi rolebasedpermissions sccm serverlesssqlpool servicenow siem sql tenablesc dedicatedsqlpool active-directory azuremlstudio machine-learning
Language:TSQL 35
apiary
ExpediaGroup / apiary
Apiary provides modules which can be combined to create a federated cloud data lake
hive datalake aws hive-metastore
35