data-engineering

There are 414 repositories under data-engineering topic.

superset
apache / superset
Apache Superset is a Data Visualization and Data Exploration Platform
superset apache apache-superset data-visualization data-viz analytics business-intelligence data-science data-engineering asf bi business-analytics data-analytics data-analysis python react sql-editor flask
Language:TypeScript 60864
Made-With-ML
GokuMohandas / Made-With-ML
Learn how to design, develop, deploy and iterate on production-grade ML applications.
machine-learning deep-learning pytorch natural-language-processing data-science python mlops data-engineering data-quality distributed-ml llms ray distributed-training
Language:Jupyter Notebook 36684
airflow
apache / airflow
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
airflow apache apache-airflow automation dag data-engineering data-integration data-orchestrator data-pipelines data-science elt etl machine-learning mlops orchestration python scheduler workflow workflow-engine workflow-orchestration
Language:Python 35634
applied-ml
eugeneyan / applied-ml
📚 Papers & tech blogs by companies sharing their work on data science & machine learning in production.
applied-data-science applied-machine-learning computer-vision data-discovery data-engineering data-quality data-science deep-learning machine-learning natural-language-processing production recsys reinforcement-learning search
26855
DataTalksClub / data-engineering-zoomcamp
Free Data Engineering course!
data-engineering dbt docker kafka prefect spark
Language:Jupyter Notebook 23905
PrefectHQ / prefect
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
python workflow data-engineering data-science workflow-engine prefect infrastructure ml-ops data-ops automation orchestration data observability pipeline
Language:Python 15502
airbytehq / airbyte
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
data pipeline data-analysis data-engineering java python etl change-data-capture data-collection data-integration elt bigquery redshift snowflake data-pipeline mssql mysql postgresql s3 self-hosted
Language:Python 14974
argoproj / argo-workflows
Workflow Engine for Kubernetes
workflow kubernetes argo dag knative airflow machine-learning argo-workflows workflow-engine hacktoberfest cloud-native cncf k8s gitops mlops batch-processing data-engineering pipelines
Language:Go 14651
andkret / Cookbook
The Data Engineering Cookbook
data-engineer data-engineering big-data best-practices cookbook
13397
data-engineer-roadmap
datastacktv / data-engineer-roadmap
Roadmap to becoming a data engineer in 2021
cloud data-engineer-roadmap data-engineering roadmap
12224
dagster-io / dagster
An orchestration platform for the development, production, and observation of data assets.
data-pipelines dagster workflow data-science workflow-automation python scheduler data-orchestrator etl analytics data-engineering mlops orchestration data-integration metadata
Language:Python 10843
Avaiga / taipy
Turns Data and AI algorithms into production-ready web applications in no time.
automation data-engineering data-integration data-ops data-visualization datascience developer-tools hacktoberfest hacktoberfest2023 job-scheduler mlops orchestration pipeline pipelines python scenario scenario-analysis taipy-core taipy-gui workflow
Language:Python 10089
great-expectations / great_expectations
Always know what to expect from your data.
pipeline-tests dataquality datacleaning datacleaner data-science data-profiling pipeline pipeline-testing cleandata dataunittest data-unit-tests eda exploratory-data-analysis exploratory-analysis exploratorydataanalysis data-quality data-engineering pipeline-debt data-profilers mlops
Language:Python 9702
xonsh
xonsh / xonsh
:shell: Python-powered shell. Full-featured and cross-platform.
artificial-intelligence bash chatgpt cli command-line console data-engineering data-science devops fish hacktoberfest iterm2 python raspberry-pi security-automation shell windows-terminal xonsh zsh
Language:Python 8207
connect
redpanda-data / connect
Fancy stream processing made operationally mundane
message-queue stream-processing streaming-data message-bus logs stream-processor cqrs event-sourcing go golang kafka amqp rabbitmq nats etl data-engineering data-ops
Language:Go 8013
mage-ai / mage-ai
🧙 Build, run, and manage data pipelines for integrating and transforming data.
machine-learning artificial-intelligence data data-engineering data-science python elt etl pipelines data-pipelines orchestration data-integration sql spark dbt pipeline reverse-etl transformation
Language:Python 7509
kestra
kestra-io / kestra
Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.
workflow workflow-engine orchestration scheduler data-pipeline elt etl data data-engineering data-orchestration data-orchestrator low-code data-integration reverse-etl data-quality pipeline
Language:Java 7170
risingwave
risingwavelabs / risingwave
SQL stream processing, analytics, and management. We decouple storage and compute to offer efficient joins, instant failover, dynamic scaling, speedy bootstrapping, and concurrent query serving.
analytics big-data cloud-native data-engineering database distributed-database etl flink kafka ksqldb materialized-view postgres postgresql real-time real-time-analytics rust serverless spark-streaming sql stream-processing
Language:Rust 6644
growthbook
growthbook / growthbook
Open Source Feature Flagging and A/B Testing Platform
abtesting statistics abtest experimentation split-testing mixpanel snowflake bigquery redshift clickhouse analytics ab-testing feature-flags feature-flagging remote-config continuous-delivery data-analysis data-science data-engineering
Language:TypeScript 5860
cloudquery
cloudquery / cloudquery
The open source high performance ELT framework powered by Apache Arrow
aws gcp azure sql data-integration elt etl etl-framework bigquery data-collection data-engineering kubernetes data airbyte github-api data-analysis google go cspm attack-surface-management
Language:Go 5728
feast-dev / feast
The Open Source Feature Store for Machine Learning
machine-learning features ml big-data feature-store python mlops data-engineering data-science data-quality
Language:Python 5403
lakeFS
treeverse / lakeFS
lakeFS - Data version control for your data lake | Git for data
data-engineering data-versioning go object-storage data-lake aws-s3 data-quality azure-blob-storage google-cloud-storage golang git-for-data lakefs apache-spark apache-sparksql hadoop-filesystem datalake datalakes data-version-control azure-storage
Language:Go 4244
whoiskatrin / sql-translator
SQL Translator is a tool for converting natural language queries into SQL code using artificial intelligence. This project is 100% free and open source.
data-analysis data-engineering dataquery datascience dataset openai postgresql query sql
Language:TypeScript 4108
rudder-server
rudderlabs / rudder-server
Privacy and Security focused Segment-alternative, in Golang and React
privacy warehouse-management data-warehouse customer-data customer-data-pipeline customer-data-platform customer-data-lake segment-alternative data-integration data-synchronization etl bigquery redshift snowflake data-pipeline elt data-engineering cdp warehouse-native event-streaming
Language:Go 4016
evidence-dev / evidence
Business intelligence as code: build fast, interactive data visualizations in pure SQL and markdown
analytics sql business-intelligence data-visualization dbt duckdb tailwindcss self-hosted open-source svelte dashboard data-engineering data-science exploratory-data-analysis finance statistics webassembly
Language:JavaScript 3873
aws-sdk-pandas
aws / aws-sdk-pandas
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
amazon-athena amazon-sagemaker-notebook apache-arrow apache-parquet athena aws aws-glue aws-lambda data-engineering data-science emr etl glue-catalog lambda modin mysql pandas python ray redshift
Language:Python 3857
God-Level-AI
hemansnation / God-Level-AI
A collection of scientific methods, processes, algorithms, and systems to build stories & models.
python datastructures numpy pandas matplotlib scikit-learn machine-learning deep-learning pytorch data-science computer-vision data-structures-and-algorithms data-visualization mlops natural-language-processing statistics tableau data-engineering data-system-design
Language:Jupyter Notebook 3507
ploomber / ploomber
The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️
workflow machine-learning data-science data-engineering mlops papermill jupyter jupyter-notebooks pipelines vscode pycharm notebooks
Language:Python 3445
adilkhash / Data-Engineering-HowTo
A list of useful resources to learn Data Engineering from scratch
distributed-systems data-engineering data-pipeline cloud-providers scala
3381
memphis
superstreamlabs / memphis
Memphis.dev is a highly scalable and effortless data streaming platform
data data-stream-processing data-streaming kubernetes messaging-queue data-engineering data-pipeline golang enrichment message-broker message-bus message-queue microservices schema-registry
Language:Go 3201
Moataz-Elmesmary / Data-Science-Roadmap
Data Science Roadmap from A to Z
data-analysis data-engineering data-science data-visualization deep-learning machine-learning mathematics probability python sql statistics cheatsheet cv-template interview-questions linear-algebra neural-network big-data chatgpt llms nlp
3090
datafold / data-diff
Compare tables within or across databases
database mysql postgresql snowflake rdbms trino data-engineering data-quality data-science data-quality-monitoring databricks-sql dataengineering dataquality oracle-database postgres sql dbt python data data-diffing
Language:Python 2923
mlops-course
GokuMohandas / mlops-course
Learn how to design, develop, deploy and iterate on production-grade ML applications.
machine-learning deep-learning pytorch mlops data-engineering data-quality data-science distributed-ml llms natural-language-processing python ray
Language:Jupyter Notebook 2838
quadratichq / quadratic
Quadratic | Technical Spreadsheet with Python, SQL, and AI
data data-analysis data-engineering data-science etl python quadratic spreadsheet sql wasm webgl
Language:TypeScript 2832
apache / incubator-devlake
Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.
data data-analysis data-engineering data-integration data-transfers devops domain-layer etl golang integration jira open-source user-friendly dashboard-friendly dora hacktoberfest
Language:Go 2520
qsv
jqnatividad / qsv
CSVs sliced, diced & analyzed.
csv data-wrangling tsv cli opendata data-engineering ckan excel luau datapackage parquet postgresql sqlite polars snappy rust sql geocode timeseries
Language:Rust 2346