data-engineering

There are 471 repositories under data-engineering topic.

superset
apache / superset
Apache Superset is a Data Visualization and Data Exploration Platform
superset apache apache-superset data-visualization data-viz analytics business-intelligence data-science data-engineering asf bi business-analytics data-analytics data-analysis python react sql-editor flask
Language:TypeScript 68064
Made-With-ML
GokuMohandas / Made-With-ML
Learn how to design, develop, deploy and iterate on production-grade ML applications.
machine-learning deep-learning pytorch natural-language-processing data-science python mlops data-engineering data-quality distributed-ml llms ray distributed-training
Language:Jupyter Notebook 43006
airflow
apache / airflow
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
airflow apache apache-airflow python scheduler workflow automation dag data-engineering data-integration data-orchestrator data-pipelines data-science elt etl machine-learning mlops orchestration workflow-engine workflow-orchestration
Language:Python 42367
DataTalksClub / data-engineering-zoomcamp
Data Engineering Zoomcamp is a free nine-week course that covers the fundamentals of data engineering.
data-engineering kafka spark dbt docker kestra
Language:Jupyter Notebook 32711
applied-ml
eugeneyan / applied-ml
📚 Papers & tech blogs by companies sharing their work on data science & machine learning in production.
applied-machine-learning production applied-data-science machine-learning data-science reinforcement-learning data-engineering recsys search deep-learning data-quality data-discovery computer-vision natural-language-processing
28302
PrefectHQ / prefect
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
automation data data-engineering data-ops data-science infrastructure ml-ops observability orchestration pipeline prefect python workflow workflow-engine
Language:Python 20354
airbytehq / airbyte
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
bigquery change-data-capture data data-analysis data-collection data-engineering data-integration data-pipeline elt etl java mssql mysql pipeline postgresql python redshift s3 self-hosted snowflake
Language:Python 19513
taipy
Avaiga / taipy
Turns Data and AI algorithms into production-ready web applications in no time.
automation data-engineering data-ops data-visualization datascience developer-tools mlops orchestration pipeline pipelines python taipy-gui workflow taipy-core hacktoberfest hacktoberfest2023 data-integration job-scheduler scenario scenario-analysis
Language:Python 18634
argoproj / argo-workflows
Workflow Engine for Kubernetes
workflow kubernetes argo dag knative airflow machine-learning argo-workflows workflow-engine hacktoberfest cloud-native cncf k8s gitops mlops batch-processing data-engineering pipelines
Language:Go 16019
andkret / Cookbook
The Data Engineering Cookbook
best-practices big-data cookbook data-engineer data-engineering
Language:Python 14509
dagster-io / dagster
An orchestration platform for the development, production, and observation of data assets.
data-pipelines dagster workflow data-science workflow-automation python scheduler data-orchestrator etl analytics data-engineering mlops orchestration data-integration metadata
Language:Python 13997
data-engineer-roadmap
datastacktv / data-engineer-roadmap
Roadmap to becoming a data engineer in 2021
data-engineer-roadmap data-engineering cloud roadmap
12694
great-expectations / great_expectations
Always know what to expect from your data.
pipeline-tests dataquality datacleaning datacleaner data-science data-profiling pipeline pipeline-testing cleandata dataunittest data-unit-tests eda exploratory-data-analysis exploratory-analysis exploratorydataanalysis data-quality data-engineering pipeline-debt data-profilers mlops
Language:Python 10741
xonsh
xonsh / xonsh
:shell: Python-powered shell. Full-featured and cross-platform.
python xonsh shell cli command-line console bash devops fish zsh iterm2 data-science data-engineering security-automation raspberry-pi artificial-intelligence
Language:Python 8971
mage-ai / mage-ai
🧙 Build, run, and manage data pipelines for integrating and transforming data.
machine-learning artificial-intelligence data data-engineering data-science python elt etl pipelines data-pipelines orchestration data-integration sql spark dbt pipeline reverse-etl transformation
Language:Python 8464
connect
redpanda-data / connect
Fancy stream processing made operationally mundane
message-queue stream-processing streaming-data message-bus logs stream-processor cqrs event-sourcing go golang kafka amqp rabbitmq nats etl data-engineering data-ops
Language:Go 8447
risingwave
risingwavelabs / risingwave
Real-time event streaming platform. Streaming CDC, stream processing, low-latency serving, and Iceberg management.
database stream-processing rust postgresql kafka materialized-view data-engineering apache-iceberg elt-pipeline etl-pipeline
Language:Rust 8338
growthbook
growthbook / growthbook
Open Source Feature Flagging and A/B Testing Platform
abtesting statistics abtest experimentation split-testing mixpanel snowflake bigquery redshift clickhouse analytics ab-testing feature-flags feature-flagging remote-config continuous-delivery data-analysis data-science data-engineering
Language:TypeScript 6856
feast-dev / feast
The Open Source Feature Store for AI/ML
big-data data-engineering data-quality data-science feature-store features machine-learning ml mlops python
Language:Python 6333
cloudquery
cloudquery / cloudquery
The open source ELT framework powered by Apache Arrow
aws gcp azure sql data-integration elt etl etl-framework bigquery data-collection data-engineering kubernetes data airbyte github-api data-analysis google go cspm attack-surface-management
Language:Go 6205
evidence-dev / evidence
Business intelligence as code: build fast, interactive data visualizations in SQL and markdown
analytics sql business-intelligence data-visualization dbt duckdb tailwindcss self-hosted svelte dashboard data-engineering data-science exploratory-data-analysis webassembly
Language:JavaScript 5551
lakeFS
treeverse / lakeFS
lakeFS - Data version control for your data lake | Git for data
data-engineering data-versioning go object-storage data-lake aws-s3 data-quality azure-blob-storage google-cloud-storage golang git-for-data lakefs apache-spark apache-sparksql hadoop-filesystem datalake datalakes data-version-control azure-storage
Language:Go 4877
whoiskatrin / sql-translator
SQL Translator is a tool for converting natural language queries into SQL code using artificial intelligence. This project is 100% free and open source.
data-analysis data-engineering dataquery datascience dataset openai postgresql query sql
Language:TypeScript 4292
rudder-server
rudderlabs / rudder-server
Privacy and Security focused Segment-alternative, in Golang and React
privacy warehouse-management data-warehouse customer-data customer-data-pipeline customer-data-platform customer-data-lake segment-alternative data-integration data-synchronization etl bigquery redshift snowflake data-pipeline elt data-engineering cdp warehouse-native event-streaming
Language:Go 4263
dlt-hub / dlt
data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
data python data-engineering data-lake data-loading data-warehouse elt extract load transform
Language:Python 4149
aws-sdk-pandas
aws / aws-sdk-pandas
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
amazon-athena amazon-sagemaker-notebook apache-arrow apache-parquet athena aws aws-glue aws-lambda data-engineering data-science emr etl glue-catalog lambda modin mysql pandas python ray redshift
Language:Python 4062
Moataz-Elmesmary / Data-Science-Roadmap
Data Science Roadmap from A to Z
data-analysis data-engineering data-science data-visualization deep-learning machine-learning mathematics probability python sql statistics cheatsheet cv-template interview-questions linear-algebra neural-network big-data chatgpt llms nlp
3938
adilkhash / Data-Engineering-HowTo
A list of useful resources to learn Data Engineering from scratch
distributed-systems data-engineering data-pipeline cloud-providers scala
3871
quadratichq / quadratic
Spreadsheet with AI, Code, Connections
ai data data-analysis data-engineering data-science etl python quadratic spreadsheet sql wasm webgl
Language:Rust 3826
ploomber / ploomber
The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️
workflow machine-learning data-science data-engineering mlops papermill jupyter jupyter-notebooks pipelines vscode pycharm notebooks
Language:Python 3601
god-level-ai
hemansnation / god-level-ai
A collection of scientific methods, processes, algorithms, and systems to build stories & models.
computer-vision data-engineering data-science data-structures-and-algorithms data-system-design data-visualization datastructures deep-learning machine-learning matplotlib mlops natural-language-processing numpy pandas python pytorch scikit-learn statistics tableau
Language:Jupyter Notebook 3560
memphis
superstreamlabs / memphis
Memphis.dev is a highly scalable and effortless data streaming platform
data data-stream-processing data-streaming kubernetes messaging-queue data-engineering data-pipeline golang enrichment message-broker message-bus message-queue microservices schema-registry
Language:Go 3326
mlops-course
GokuMohandas / mlops-course
Learn how to design, develop, deploy and iterate on production-grade ML applications.
data-engineering data-quality data-science deep-learning distributed-ml llms machine-learning mlops natural-language-processing python pytorch ray
Language:Jupyter Notebook 3199
qsv
dathere / qsv
Blazing-fast Data-Wrangling toolkit
csv data-wrangling opendata data-engineering ckan excel luau parquet postgresql sqlite polars sql geocode timeseries dcat metadata statistics fair-data sampling ai
Language:Rust 3150
datafold / data-diff
Compare tables within or across databases
database mysql postgresql snowflake rdbms trino data-engineering data-quality data-science data-quality-monitoring databricks-sql dataengineering dataquality oracle-database postgres sql dbt python data data-diffing
Language:Python 2989
apache / incubator-devlake
Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.
data data-analysis data-engineering data-integration data-transfers devops domain-layer etl golang integration jira open-source user-friendly dashboard-friendly dora hacktoberfest
Language:Go 2811