data-orchestration

There are 5 repositories under data-orchestration topic.

• Alluxio / alluxio

  Alluxio, data orchestration for analytics and machine learning in the cloud

  alluxiomemory-speedhadoopsparkprestotensorflowdata-analysisdata-orchestrationvirtual-distributed-filesystem
  Language:Java 6631
• kestra

  kestra-io / kestra

  Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.

  workflowworkflow-engineorchestrationschedulerdata-pipelineeltetldatadata-engineeringdata-orchestrationdata-orchestratorlow-codedata-integrationreverse-etldata-qualitypipeline
  Language:Java 6264

• cubefs / cubefs

  cloud-native file store

  distributed-storagecncfkubernetescloud-storageerasure-codinghybrid-cloudai-native-storageobject-storagecloud-native-storagedistributed-file-systemdata-orchestrationfuse
  Language:Go 4310

• apache / incubator-graphar

  An open source, standard data file format for graph data storage and retrieval.

  big-datagraphgraph-storagedata-orchestrationetlgraph-analysispysparkspark
  Language:C++ 173
• Skytrax-Data-Warehouse

  iam-mhaseeb / Skytrax-Data-Warehouse

  A full data warehouse infrastructure with ETL pipelines running inside docker on Apache Airflow for data orchestration, AWS Redshift for cloud data warehouse and Metabase to serve the needs of data visualizations such as analytical dashboards.

  pythonpython3databasedata-visualizationdata-analysisdata-warehousedata-warehousingredshiftairflowdockersqldata-orchestrationdata-analyticss3s3-bucketmetabasedata-engineeringdata-cleaningdata-processing
  Language:Python 131

• jonathanneo / data-aware-orchestration

  Data-aware orchestration with dagster, dbt, and airbyte

  data-orchestration
  Language:Python 28

• ozkary / data-engineering-mta-turnstile

  Data Engineering - Metropolitan Transportation Authority (MTA) Subway Data Analysis

  data-lakedockeranalysisdata-engineeringpythonsqlstatic-analysisvscodebigquerydata-modelingdata-orchestrationdata-pipelinedata-warehousedbtjupyter-notebookprefectterraform
  Language:Jupyter Notebook 19

• SAP-samples / btp-data-to-value-workshop

  This repo contains a dataset, exercises, and sample code for an end-to-end SAP BTP data-to-value bootcamp covering SAP HANA Cloud, SAP Data Warehouse Cloud, SAP Data Intelligence Cloud, and SAP Analytics Cloud.

  samplesample-codeworkshopsap-btpsap-analytics-cloudsap-data-warehouse-cloudsap-data-intelligence-cloudmachine-learningsap-hana-clouddata-managementdata-to-valueanalyticspredictive-planningdata-sciencedata-orchestrationadvanced-analytics
  Language:Jupyter Notebook 18

• astronomer / airflow-provider-fivetran-async

  A new Airflow Provider for Fivetran, maintained by Astronomer and Fivetran

  airflowairflow-operatorairflow-providerapache-airflowdagdata-orchestrationetlpythonworkflow
  Language:Python 15

• Alluxio / k8s-operator

  An operator for managing Alluxio system on Kubernetes cluster

  alluxiodata-analysisdata-orchestrationkuberneteskubernetes-operatormachine-learning
  Language:Go 10

• kestra-io / examples

  Best practices for data workflows, integrations with the Modern Data Stack (MDS), Infrastructure as Code (IaC), Cloud Provider Services

  analytics-engineeringautomationdata-engineeringdata-orchestrationdata-pipelinesdata-workflowsorchestration
  Language:HCL 9

• anna-geller / kestra-ci-cd

  CI/CD repository template to automate deployments of your production flows

  automationdata-engineeringdata-orchestrationdata-pipelinesdata-workflowsorchestration
  Language:HCL 5

• anna-geller / kestra-terraform-examples

  Bring Infrastructure as Code best practices to your data workflows with Kestra and Terraform

  automationawsdata-engineeringdata-infrastructuredata-orchestrationinfrastructure-as-codekestraplatform-engineeringterraformworkflow-as-codeworkflow-orchestration
  Language:HCL 5

• stemitom / postgres-pipeline

  A simple pipeline infrastructure with ETL pipeline contained in a Docker environment on Apache Airflow for orchestration and Postgres for data warehousing

  pythondockerairflowsqldatabasedata-engineeringdata-warehousingetletl-pipelinepostgresdata-orchestration
  Language:Python 4

• dagster-io / dagster-quickstart

  Get started with Dagster ASAP

  dagsterdata-orchestrationlearning-by-doing
  Language:Python 2

• zpencerguy / superdoppler

  Data orchestration repo with Docker deployment

  data-orchestrationdata-pipelinespython
  Language:Python 2
• azure-data-factory-data-vault

  Annielytix / azure-data-factory-data-vault

  Working with SCD Type (Change Data Capture) and need a Data Vault model to test Azure Data Factory v2? - This Code with Help!

  azuredata-factorydata-vaultscd-type-2change-data-captureadf-v2adforchetrationorchestrationorchestration-frameworkcloud-migrationdata-orchestration
  1

• Data-Practitioner / Crypto-Data-Metric-Project

  Combine all four layers of analytics (descriptive, diagnostic, predictive, and prescriptive) on crypto data in a dashboard format as a web application.

  awsdata-analysisdata-miningdata-orchestrationdata-storytellingdata-transformationdata-visualizationdatabase-designdockeretlml-deploymentnlppostgrespower-bipythonsqltext-miningtime-series-forecastingweb-applicationweb-scraping
  Language:Jupyter Notebook 1

• jasontanx / prefect-learning

  Prefect - Data orchestration tool practice & learning

  data-engineerdata-orchestrationprefectworkflow-management
  Language:Python 0
• data-duck-pond

  tanega / data-duck-pond

  A poor-man's data lake fill with ducks

  dagsterdata-analyticsdata-orchestrationdata-processingdata-visualizationdatalakedockerdocker-composeduckdbgeospatial-analysisgeospatial-processingpythontraefik-dockertraefik-v2
  Language:Python 0

• ddeutils / ddeapp-airflow

  Data Developer & Engineer App with Airflow

  airflowdata-engineeringdata-orchestration

• jacquessham / airflow_notes

  Repository to store scripts and notes on Airflow

  airflowdata-orchestrationetlpython
  Language:Python

• MostafaNabilll / end2end_pipeline

  End to End data engineering project

  airflowdata-modelingdata-orchestrationdata-pipelinedata-transformationdata-visualizationdata-warehousingpython
  Language:Python