apache-airflow

There are 16 repositories under apache-airflow topic.

• airflow

  apache / airflow

  Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

  airflowapacheapache-airflowpythonschedulerworkflowautomationdagdata-engineeringdata-integrationdata-orchestratordata-pipelinesdata-scienceeltetlmachine-learningmlopsorchestrationworkflow-engineworkflow-orchestration
  Language:Python 34692

• jghoman / awesome-apache-airflow

  Curated list of resources about Apache Airflow

  airflowapache-airflowworkflow-managementawesomeawesome-list
  Language:Shell 3577

• elyra-ai / elyra

  Elyra extends JupyterLab with an AI centric approach.

  notebooksnotebook-jupyterjupyterlabjupyterlab-extensionsaimachine-learningpipelineskubeflow-pipelinespythonelyrabinderjupyterlab-notebookskubeflowhacktoberfestjupyterlab-extensionanacondapypidockerapache-airflowairflow
  Language:Python 1780

• teamclairvoyant / airflow-maintenance-dags

  A series of DAGs/Workflows to help maintain the operation of Airflow

  airflowdagworkflowairflow-maintenance-dagsmaintenancecleanupapache-airflow
  Language:Python 1603
• goodreads_etl_pipeline

  san089 / goodreads_etl_pipeline

  An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

  etl-pipelineetl-frameworksparkapache-sparkapache-airflowairflowredshiftemr-clusterlivys3warehousedata-lakeschedulerdata-migrationdata-engineeringdata-engineering-pipelinepythongoodreads-data-pipelineairflow-dagetl-job
  Language:Python 1234

• ajbosco / dag-factory

  Dynamically generate Apache Airflow DAGs from YAML configuration files

  airflowapache-airflowdagspython
  Language:Python 1081

• tuanavu / airflow-tutorial

  Apache Airflow tutorial

  apache-airflowpython
  Language:Jupyter Notebook 893

• couler-proj / couler

  Unified Interface for Constructing and Managing Workflows on different workflow engines, such as Argo Workflows, Tekton Pipelines, and Apache Airflow.

  cloud-nativeworkflow-managementmachine-learningdistributed-computingkubernetesworkflow-engineargo-workflowstekton-pipelinesapache-airflowunified-interfaceunified-apischedulerworkflow-automationkubeflowpython
  Language:Python 891

• astronomer / astronomer-cosmos

  Run your dbt Core projects as Apache Airflow DAGs and Task Groups with a few lines of code

  airflowairflow-operatorsapache-airflowdbtpythonworkflow
  Language:Python 463

• astronomer / astronomer

  Helm Charts for the Astronomer Platform, Apache Airflow as a Service on Kubernetes

  apache-airflowkubernetesdockerastronomer-platform
  Language:Python 452

• blockchain-etl / ethereum-etl-airflow

  Airflow DAGs for exporting, loading, and parsing the Ethereum blockchain data. How to get any Ethereum smart contract into BigQuery https://towardsdatascience.com/how-to-get-any-ethereum-smart-contract-into-bigquery-in-8-mins-bab5db1fdeee

  ethereumblockchain-analyticsdata-engineeringcryptocurrencyweb3apache-airflowcryptodata-analyticsetlgcpgoogle-cloudgoogle-cloud-platformon-chain-analysis
  Language:Python 386

• andreax79 / airflow-code-editor

  A plugin for Apache Airflow that allows you to edit DAGs in browser

  apache-airflowairflow-pluginairflowpython
  Language:JavaScript 371

• airflow-plugins / Example-Airflow-DAGs

  Example DAGs using hooks and operators from Airflow Plugins

  apache-airflowdagetlairflow-pluginsseleniumgoogle-analyticsfacebook-adssftpmarketosalesforcehubspotmarketo-sdkimapairflowmongodbmailgunsingerdbt
  Language:Python 328

• astronomer / astro-cli

  CLI that makes it easy to create, test and deploy Airflow DAGs to Astronomer

  apache-airflowastronomer-platformkubernetes
  Language:Go 325

• astronomer / astro-sdk

  Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

  pythonsqlpandasairflowsqlitebigquerypostgressnowflakegcss3etleltdagsworkflowsdata-analysisdata-scienceapache-airflow
  Language:Python 322

• teamclairvoyant / airflow-rest-api-plugin

  A plugin for Apache Airflow that exposes rest end points for the Command Line Interfaces

  airflowrest-apipluginapache-airflowairflow-webserverairflow-plugin
  Language:Python 322

• apache / airflow-client-python

  Apache Airflow - OpenApi Client for Python

  airflowapacheapache-airflowapache-airflow-clientpython
  Language:Python 319

• GoogleCloudPlatform / airflow-operator

  Kubernetes custom controller and CRDs to managing Airflow

  kuberneteskubernetes-operatorapache-airflowairflowcrdkubernetes-controllerworkflow-engineairflow-operator
  Language:Go 299

• astronomer / airflow-chart

  A Helm chart to install Apache Airflow on Kubernetes

  helm-chartairflowapache-airflowkubernetes
  Language:Python 267

• timkpaine / paperboy

  A web frontend for scheduling Jupyter notebook reports

  jupyterjupyterlabjupyter-notebookspapermillnteractairflowapache-airflowphosphorjsdaskscheduling-notebooksjupyter-notebooknotebookdockerkubernetesluigicelery
  Language:Python 248

• teamclairvoyant / airflow-scheduler-failover-controller

  A process that runs in unison with Apache Airflow to control the Scheduler process to ensure High Availability

  airflowscheduler-failover-controllerapache-airflowairflow-schedulerairflow-clusterhigh-availability
  Language:Python 232

• xnuinside / airflow_in_docker_compose

  Apache Airflow in Docker Compose (for both versions 1.10.* and 2.*)

  docker-compose-filesairflowdocker-airflowcelery-executordockerdocker-composedocker-compose-templateapache-airflow
  Language:Python 185

• kaxil / airflowctl

  A CLI tool to streamline getting started with Apache Airflow™ and managing multiple Airflow projects

  airflowapache-airflowclidagspython
  Language:Python 170

• rolanddb / airflow-on-kubernetes

  A guide to running Airflow on Kubernetes

  apache-airflowkubernetes
  167

• apache / airflow-client-go

  Apache Airflow - OpenApi Client for Go

  airflowapacheapache-airflowapache-airflow-clientgo
  Language:Go 159

• astronomer / astronomer-providers

  Airflow Providers containing Deferrable Operators & Sensors from Astronomer

  pythonapache-airflowairflowworkflowairflow-operatorsairflow-providers
  Language:Python 128

• ninja-van / airflow-boilerplate

  A complete development environment setup for working with Airflow

  airflowpythonapache-airflowpycharmboilerplate
  Language:Python 122

• datacamp / viewflow

  Viewflow is an Airflow-based framework that allows data scientists to create data models without writing Airflow code.

  airflowdata-sciencepackagespythonapache-airflowworkflowdata-engineering
  Language:Python 121

• airscholar / e2e-data-engineering

  An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. All components are containerized with Docker for easy deployment and scalability.

  apache-airflowapache-kafkaapache-sparkapache-zookeeperbig-datacassandracontainerizationdata-engineeringdata-pipelinedata-processingdata-storagedockeretl-pipelinepostgresqlreal-time-analytics
  Language:Python 109

• caoergou / airflow-extended-api-plugin

  Yet another Airflow plugin using CLI command as RESTful api, supports Airflow v2.X.

  airflow-pluginairflowapache-airflowrestful-apicommand-lineapi
  Language:Python 103

• Wittline / uber-expenses-tracking

  The goal of this project is to track the expenses of Uber Rides and Uber Eats through data Engineering processes using technologies such as Apache Airflow, AWS Redshift and Power BI.

  data-engineeringpythonapache-airflowetl-pipelineaws-redshiftpower-bidata-modelinguber-eatsuber-dataexpenses-dashboardexpenses-trackerairflow-dockeruberaws
  Language:Jupyter Notebook 98

• Toloka / dbt-af

  Distributed run of dbt models using Airflow

  airflowapache-airflowdbtpython
  Language:Python 92

• PowerDataHub / terraform-aws-airflow

  Terraform module to deploy an Apache Airflow cluster on AWS, backed by RDS PostgreSQL for metadata, S3 for logs and SQS as message broker with CeleryExecutor

  apache-airflowairflowterraformterraform-modulesawsceleryterraform-modulehacktoberfest
  Language:HCL 84

• idealista / airflow-role

  Ansible role to install Apache Airflow

  airflowansibleansible-roledebianapache-airflowairflow-role
  Language:Jinja 81

• gestaogovbr / Ro-dou

  Gerador de DAGs no Airflow para fazer clipping do Diário Oficial da União.

  diario-oficial-da-uniaoairflow-dagsgovernment-gazetteopen-datadados-abertosapache-airflow
  Language:Python 64

• airflow-plugins / airflow_api_plugin

  REST-like API exposing Airflow data and operations

  apache-airflow
  Language:Python 60