data-engineering

There are 514 repositories under data-engineering topic.

superset
apache / superset
Apache Superset is a Data Visualization and Data Exploration Platform
analytics apache apache-superset asf bi business-analytics business-intelligence data-analysis data-analytics data-engineering data-science data-visualization data-viz flask python react sql-editor superset
Language:TypeScript 68856
Made-With-ML
GokuMohandas / Made-With-ML
Learn how to design, develop, deploy and iterate on production-grade ML applications.
data-engineering data-quality data-science deep-learning distributed-ml distributed-training llms machine-learning mlops natural-language-processing python pytorch ray
Language:Jupyter Notebook 44206
airflow
apache / airflow
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
airflow apache apache-airflow python scheduler workflow automation dag data-engineering data-integration data-orchestrator data-pipelines data-science elt etl machine-learning mlops orchestration workflow-engine workflow-orchestration
Language:Python 43122
DataTalksClub / data-engineering-zoomcamp
Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. The next cohort starts in January 2026. Join the course here 👇🏼
course data-engineering dbt docker free kafka kestra spark
Language:Jupyter Notebook 33378
applied-ml
eugeneyan / applied-ml
📚 Papers & tech blogs by companies sharing their work on data science & machine learning in production.
applied-machine-learning production applied-data-science machine-learning data-science reinforcement-learning data-engineering recsys search deep-learning data-quality data-discovery computer-vision natural-language-processing
28495
PrefectHQ / prefect
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
automation data data-engineering data-ops data-science infrastructure ml-ops observability orchestration pipeline prefect python workflow workflow-engine
Language:Python 20772
airbytehq / airbyte
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
bigquery change-data-capture data data-analysis data-collection data-engineering data-integration data-pipeline elt etl java mssql mysql pipeline postgresql python redshift s3 self-hosted snowflake
Language:Python 19980
taipy
Avaiga / taipy
Turns Data and AI algorithms into production-ready web applications in no time.
automation data-engineering data-integration data-ops data-visualization datascience developer-tools hacktoberfest hacktoberfest2023 job-scheduler mlops orchestration pipeline pipelines python scenario scenario-analysis taipy-core taipy-gui workflow
Language:Python 18849
argoproj / argo-workflows
Workflow Engine for Kubernetes
workflow kubernetes argo dag knative airflow machine-learning argo-workflows workflow-engine hacktoberfest cloud-native cncf k8s gitops mlops batch-processing data-engineering pipelines
Language:Go 16166
andkret / Cookbook
The Data Engineering Cookbook
best-practices big-data cookbook data-engineer data-engineering
Language:Python 14613
dagster-io / dagster
An orchestration platform for the development, production, and observation of data assets.
analytics dagster data-engineering data-integration data-orchestrator data-pipelines data-science etl hacktoberfest metadata mlops orchestration python scheduler workflow workflow-automation
Language:Python 14365
data-engineer-roadmap
datastacktv / data-engineer-roadmap
Roadmap to becoming a data engineer in 2021
cloud data-engineer-roadmap data-engineering roadmap
12715
great-expectations / great_expectations
Always know what to expect from your data.
pipeline-tests dataquality datacleaning datacleaner data-science data-profiling pipeline pipeline-testing cleandata dataunittest data-unit-tests eda exploratory-data-analysis exploratory-analysis exploratorydataanalysis data-quality data-engineering pipeline-debt data-profilers mlops
Language:Python 10903
xonsh
xonsh / xonsh
:shell: Python-powered shell. Full-featured and cross-platform.
artificial-intelligence bash chatgpt cli command-line console data-engineering data-science devops fish iterm2 python raspberry-pi security-automation shell xonsh zsh
Language:Python 9041
mage-ai / mage-ai
🧙 Build, run, and manage data pipelines for integrating and transforming data.
artificial-intelligence data data-engineering data-integration data-pipelines data-science dbt elt etl machine-learning orchestration pipeline pipelines python reverse-etl spark sql transformation
Language:Python 8517
connect
redpanda-data / connect
Fancy stream processing made operationally mundane
message-queue stream-processing streaming-data message-bus logs stream-processor cqrs event-sourcing go golang kafka amqp rabbitmq nats etl data-engineering data-ops
Language:Go 8493
risingwave
risingwavelabs / risingwave
Streaming data platform. Real-time stream processing, low-latency serving, and Iceberg table management.
apache-iceberg data-engineering database etl-pipeline kafka materialized-view postgresql rust stream-processing
Language:Rust 8488
growthbook
growthbook / growthbook
Open Source Feature Flagging and A/B Testing Platform
abtesting statistics abtest experimentation split-testing mixpanel snowflake bigquery redshift clickhouse analytics ab-testing feature-flags feature-flagging remote-config continuous-delivery data-analysis data-science data-engineering
Language:TypeScript 7076
feast-dev / feast
The Open Source Feature Store for AI/ML
big-data data-engineering data-quality data-science feature-store features machine-learning ml mlops python
Language:Python 6459
cloudquery
cloudquery / cloudquery
Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.
aws gcp azure sql data-integration elt etl etl-framework bigquery data-collection data-engineering kubernetes data airbyte github-api data-analysis google go cspm attack-surface-management
Language:Go 6249
evidence-dev / evidence
Business intelligence as code: build fast, interactive data visualizations in SQL and markdown
analytics sql business-intelligence data-visualization dbt duckdb tailwindcss self-hosted svelte dashboard data-engineering data-science exploratory-data-analysis webassembly
Language:JavaScript 5675
lakeFS
treeverse / lakeFS
lakeFS - Data version control for your data lake | Git for data
data-engineering data-versioning go object-storage data-lake aws-s3 data-quality azure-blob-storage google-cloud-storage golang git-for-data lakefs apache-spark apache-sparksql hadoop-filesystem datalake datalakes data-version-control azure-storage
Language:Go 4962
Daft
Eventual-Inc / Daft
Distributed query engine providing simple and reliable data processing for any modality and scale
artificial-intelligence big-data data-engineering distributed-computing machine-learning multimodal python rust
Language:Rust 4677
dlt-hub / dlt
data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
data python data-engineering data-lake data-loading data-warehouse elt extract load transform
Language:Python 4490
rudder-server
rudderlabs / rudder-server
Privacy and Security focused Segment-alternative, in Golang and React
bigquery cdp customer-data customer-data-lake customer-data-pipeline customer-data-platform data-engineering data-integration data-pipeline data-synchronization data-warehouse elt etl event-streaming privacy redshift segment-alternative snowflake warehouse-management warehouse-native
Language:Go 4314
whoiskatrin / sql-translator
SQL Translator is a tool for converting natural language queries into SQL code using artificial intelligence. This project is 100% free and open source.
data-analysis data-engineering dataquery datascience dataset openai postgresql query sql
Language:TypeScript 4305
aws-sdk-pandas
aws / aws-sdk-pandas
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
python aws pandas apache-arrow apache-parquet data-engineering etl data-science redshift athena lambda aws-lambda aws-glue emr amazon-athena glue-catalog mysql amazon-sagemaker-notebook modin ray
Language:Python 4073
Moataz-Elmesmary / Data-Science-Roadmap
Data Science Roadmap from A to Z
big-data chatgpt cheatsheet cv-template data-analysis data-engineering data-science data-visualization deep-learning interview-questions linear-algebra llms machine-learning mathematics neural-network nlp probability python sql statistics
4002
adilkhash / Data-Engineering-HowTo
A list of useful resources to learn Data Engineering from scratch
distributed-systems data-engineering data-pipeline cloud-providers scala
3907
quadratichq / quadratic
Spreadsheet with AI, Code, Connections
python data-science spreadsheet sql quadratic etl data-engineering data-analysis data wasm webgl ai
Language:Rust 3865
AI-Engineer-Headquarters
hemansnation / AI-Engineer-Headquarters
A collection of scientific methods, processes, algorithms, and systems to build stories & models.
computer-vision data-engineering data-science data-structures-and-algorithms data-system-design data-visualization datastructures deep-learning machine-learning matplotlib mlops natural-language-processing numpy pandas python pytorch scikit-learn statistics tableau
Language:Jupyter Notebook 3658
Netflix / maestro
Maestro: Netflix’s Workflow Orchestrator
analytics automation batch-processing dag data-engineering data-ops data-orchestrator data-pipelines data-science elt etl java machine-learning maestro mlops orchestration scheduler workflow workflow-engine workflow-orchestration
Language:Java 3642
ploomber / ploomber
The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️
data-engineering data-science jupyter jupyter-notebooks machine-learning mlops notebooks papermill pipelines pycharm vscode workflow
Language:Python 3603
memphis
superstreamlabs / memphis
Memphis.dev is a highly scalable and effortless data streaming platform
data data-stream-processing data-streaming kubernetes messaging-queue data-engineering data-pipeline golang enrichment message-broker message-bus message-queue microservices schema-registry
Language:Go 3407
qsv
dathere / qsv
Blazing-fast Data-Wrangling toolkit
ai ckan csv data-engineering data-wrangling dcat excel fair-data fairification geocode luau metadata opendata parquet polars postgresql sampling sql statistics timeseries
Language:Rust 3282
mlops-course
GokuMohandas / mlops-course
Learn how to design, develop, deploy and iterate on production-grade ML applications.
data-engineering data-quality data-science deep-learning distributed-ml llms machine-learning mlops natural-language-processing python pytorch ray
Language:Jupyter Notebook 3218