data-engineering

There are 389 repositories under data-engineering topic.

superset
apache / superset
Apache Superset is a Data Visualization and Data Exploration Platform
superset apache apache-superset data-visualization data-viz analytics business-intelligence data-science data-engineering asf bi business-analytics data-analytics data-analysis python react sql-editor flask
Language:TypeScript 58765
Made-With-ML
GokuMohandas / Made-With-ML
Learn how to design, develop, deploy and iterate on production-grade ML applications.
machine-learning deep-learning pytorch natural-language-processing data-science python mlops data-engineering data-quality distributed-ml llms ray distributed-training
Language:Jupyter Notebook 35633
airflow
apache / airflow
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
airflow apache apache-airflow automation dag data-engineering data-integration data-orchestrator data-pipelines data-science elt etl machine-learning mlops orchestration python scheduler workflow workflow-engine workflow-orchestration
Language:Python 34439
applied-ml
eugeneyan / applied-ml
📚 Papers & tech blogs by companies sharing their work on data science & machine learning in production.
applied-data-science applied-machine-learning computer-vision data-discovery data-engineering data-quality data-science deep-learning machine-learning natural-language-processing production recsys reinforcement-learning search
25871
DataTalksClub / data-engineering-zoomcamp
Free Data Engineering course!
data-engineering kafka spark dbt docker prefect
Language:Jupyter Notebook 22431
PrefectHQ / prefect
Prefect is a workflow orchestration tool empowering developers to build, observe, and react to data pipelines
python workflow data-engineering data-science workflow-engine prefect infrastructure ml-ops data-ops automation orchestration data observability pipeline
Language:Python 14599
argoproj / argo-workflows
Workflow Engine for Kubernetes
workflow kubernetes argo dag knative airflow machine-learning argo-workflows workflow-engine hacktoberfest cloud-native cncf k8s gitops mlops batch-processing data-engineering pipelines
Language:Go 14276
airbytehq / airbyte
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
bigquery change-data-capture data data-analysis data-collection data-engineering data-integration data-pipeline elt etl java mssql mysql pipeline postgresql python redshift s3 self-hosted snowflake
Language:Python 13970
andkret / Cookbook
The Data Engineering Cookbook
best-practices big-data cookbook data-engineer data-engineering
12919
data-engineer-roadmap
datastacktv / data-engineer-roadmap
Roadmap to becoming a data engineer in 2021
cloud data-engineer-roadmap data-engineering roadmap
11944
dagster-io / dagster
An orchestration platform for the development, production, and observation of data assets.
analytics dagster data-engineering data-integration data-orchestrator data-pipelines data-science etl metadata mlops orchestration python scheduler workflow workflow-automation
Language:Python 10194
great-expectations / great_expectations
Always know what to expect from your data.
cleandata data-engineering data-profilers data-profiling data-quality data-science data-unit-tests datacleaner datacleaning dataquality dataunittest eda exploratory-analysis exploratory-data-analysis exploratorydataanalysis mlops pipeline pipeline-debt pipeline-testing pipeline-tests
Language:Python 9450
Avaiga / taipy
Turns Data and AI algorithms into production-ready web applications in no time.
automation data-engineering data-integration data-ops data-visualization datascience developer-tools hacktoberfest hacktoberfest2023 job-scheduler mlops orchestration pipeline pipelines python scenario scenario-analysis taipy-core taipy-gui workflow
Language:Python 8380
xonsh
xonsh / xonsh
:shell: Python-powered, cross-platform, Unix-gazing shell.
bash cli command-line console data-engineering data-science devops fish hacktoberfest iterm2 prompt python python-shell script security-automation shell terminal windows-terminal xonsh zsh
Language:Python 8000
benthos
benthosdev / benthos
Fancy stream processing made operationally mundane
amqp cqrs data-engineering data-ops etl event-sourcing go golang kafka logs message-bus message-queue nats rabbitmq stream-processing stream-processor streaming-data
Language:Go 7564
mage-ai / mage-ai
🧙 Build, run, and manage data pipelines for integrating and transforming data.
artificial-intelligence data data-engineering data-integration data-pipelines data-science dbt elt etl machine-learning orchestration pipeline pipelines python reverse-etl spark sql transformation
Language:Python 7002
kestra
kestra-io / kestra
Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.
data data-engineering data-integration data-orchestration data-orchestrator data-pipeline data-quality elt etl low-code orchestration pipeline reverse-etl scheduler workflow workflow-engine
Language:Java 6318
risingwave
risingwavelabs / risingwave
Cloud-native SQL stream processing, analytics, and management. KsqlDB and Apache Flink alternative. 🚀 10x more productive. 🚀 10x more cost-efficient.
database stream-processing cloud-native sql distributed-database rust serverless postgresql real-time postgres postgresql-database flink kafka analytics big-data spark-streaming ksqldb spark materialized-view data-engineering
Language:Rust 6290
cloudquery
cloudquery / cloudquery
The open source high performance ELT framework powered by Apache Arrow
airbyte attack-surface-management aws azure bigquery cspm data data-analysis data-collection data-engineering data-integration elt etl etl-framework gcp github-api go google kubernetes sql
Language:Go 5584
growthbook
growthbook / growthbook
Open Source Feature Flagging and A/B Testing Platform
ab-testing abtest abtesting analytics bigquery clickhouse continuous-delivery data-analysis data-engineering data-science experimentation feature-flagging feature-flags mixpanel redshift remote-config snowflake split-testing statistics
Language:TypeScript 5516
feast-dev / feast
Feature Store for Machine Learning
machine-learning features ml big-data feature-store python mlops data-engineering data-science data-quality
Language:Python 5251
lakeFS
treeverse / lakeFS
lakeFS - Data version control for your data lake | Git for data
data-engineering data-versioning go object-storage data-lake aws-s3 data-quality azure-blob-storage google-cloud-storage golang git-for-data lakefs apache-spark apache-sparksql hadoop-filesystem datalake datalakes data-version-control azure-storage
Language:Go 4060
whoiskatrin / sql-translator
SQL Translator is a tool for converting natural language queries into SQL code using artificial intelligence. This project is 100% free and open source.
data-analysis data-engineering dataquery datascience dataset openai postgresql query sql
Language:TypeScript 3965
aws-sdk-pandas
aws / aws-sdk-pandas
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
amazon-athena amazon-sagemaker-notebook apache-arrow apache-parquet athena aws aws-glue aws-lambda data-engineering data-science emr etl glue-catalog lambda modin mysql pandas python ray redshift
Language:Python 3800
God-Level-Data-Science-ML-Full-Stack
hemansnation / God-Level-Data-Science-ML-Full-Stack
A collection of scientific methods, processes, algorithms, and systems to build stories & models. Whether you are a fresher in the field or an experienced professional who wants to transition into Data Science & AI
python datastructures numpy pandas matplotlib scikit-learn machine-learning deep-learning pytorch data-science computer-vision data-structures-and-algorithms data-visualization mlops natural-language-processing statistics tableau data-engineering data-system-design
Language:Jupyter Notebook 3470
ploomber / ploomber
The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️
workflow machine-learning data-science data-engineering mlops papermill jupyter jupyter-notebooks pipelines vscode pycharm notebooks
Language:Python 3372
evidence-dev / evidence
Business intelligence as code: build fast, interactive data visualizations in pure SQL and markdown
analytics sql business-intelligence data-visualization dbt duckdb tailwindcss self-hosted open-source svelte dashboard data-engineering data-science exploratory-data-analysis finance statistics webassembly
Language:JavaScript 3312
memphis
memphisdev / memphis
Memphis.dev is a highly scalable and effortless data streaming platform
data data-stream-processing data-streaming kubernetes messaging-queue data-engineering data-pipeline golang enrichment message-broker message-bus message-queue microservices schema-registry
Language:Go 3148
adilkhash / Data-Engineering-HowTo
A list of useful resources to learn Data Engineering from scratch
distributed-systems data-engineering data-pipeline cloud-providers scala
3128
Moataz-Elmesmary / Data-Science-Roadmap
Data Science Roadmap from A to Z
big-data chatgpt cheatsheet cv-template data-analysis data-engineering data-science data-visualization deep-learning interview-questions linear-algebra llms machine-learning mathematics neural-network nlp probability python sql statistics
2837
datafold / data-diff
Compare tables within or across databases
data data-diffing data-engineering data-quality data-quality-monitoring data-science database databricks-sql dataengineering dataquality dbt mysql oracle-database postgres postgresql python rdbms snowflake sql trino
Language:Python 2836
mlops-course
GokuMohandas / mlops-course
Learn how to design, develop, deploy and iterate on production-grade ML applications.
machine-learning deep-learning pytorch mlops data-engineering data-quality data-science distributed-ml llms natural-language-processing python ray
Language:Jupyter Notebook 2739
quadratichq / quadratic
Quadratic | Data Science Spreadsheet with Python & SQL
data data-analysis data-engineering data-science etl python quadratic spreadsheet sql wasm webgl
Language:TypeScript 2712
apache / incubator-devlake
Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.
data data-analysis data-engineering data-integration data-transfers devops domain-layer etl golang integration jira open-source user-friendly dashboard-friendly dora hacktoberfest
Language:Go 2423
qsv
jqnatividad / qsv
CSVs sliced, diced & analyzed.
ckan cli csv data-engineering data-wrangling datapackage excel geocode luau opendata parquet polars postgresql rust snappy sql sqlite timeseries tsv
Language:Rust 2220
metarank / metarank
A low code Machine Learning personalized ranking service for articles, listings, search results, recommendations that boosts user engagement. A friendly Learn-to-Rank engine
automl data-engineering data-science deep-learning feature-engineering feature-extraction kubernetes machine-learning neural-networks personalization ranking scala search
Language:Scala 1982