apache-airflow

There are 18 repositories under apache-airflow topic.

• airflow

  apache / airflow

  Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

  airflowapacheapache-airflowautomationdagdata-engineeringdata-integrationdata-orchestratordata-pipelinesdata-scienceeltetlmachine-learningmlopsorchestrationpythonschedulerworkflowworkflow-engineworkflow-orchestration
  Language:Python 43128

• jghoman / awesome-apache-airflow

  Curated list of resources about Apache Airflow

  airflowapache-airflowawesomeawesome-listworkflow-management
  Language:Shell 3853

• elyra-ai / elyra

  Elyra extends JupyterLab with an AI centric approach.

  notebooksnotebook-jupyterjupyterlabjupyterlab-extensionsaimachine-learningpipelineskubeflow-pipelinespythonelyrabinderjupyterlab-notebookskubeflowhacktoberfestjupyterlab-extensionanacondapypidockerapache-airflowairflow
  Language:Python 1969

• teamclairvoyant / airflow-maintenance-dags

  A series of DAGs/Workflows to help maintain the operation of Airflow

  airflowairflow-maintenance-dagsapache-airflowcleanupdagmaintenanceworkflow
  Language:Python 1754
• goodreads_etl_pipeline

  san089 / goodreads_etl_pipeline

  An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

  etl-pipelineetl-frameworksparkapache-sparkapache-airflowairflowredshiftemr-clusterlivys3warehousedata-lakeschedulerdata-migrationdata-engineeringdata-engineering-pipelinepythongoodreads-data-pipelineairflow-dagetl-job
  Language:Python 1430

• astronomer / dag-factory

  Construct Apache Airflow DAGs Declaratively via YAML configuration files

  airflowapache-airflowdagspython
  Language:Python 1369

• astronomer / astronomer-cosmos

  Run your dbt Core or dbt Fusion projects as Apache Airflow DAGs and Task Groups with a few lines of code

  airflowairflow-operatorsapache-airflowdbtpythonworkflow
  Language:Python 1080

• tuanavu / airflow-tutorial

  Apache Airflow tutorial

  apache-airflowpython
  Language:Jupyter Notebook 968

• couler-proj / couler

  Unified Interface for Constructing and Managing Workflows on different workflow engines, such as Argo Workflows, Tekton Pipelines, and Apache Airflow.

  cloud-nativeworkflow-managementmachine-learningdistributed-computingkubernetesworkflow-engineargo-workflowstekton-pipelinesapache-airflowunified-interfaceunified-apischedulerworkflow-automationkubeflowpython
  Language:Python 943

• astronomer / astronomer

  Helm Charts for the Astronomer Platform, Apache Airflow as a Service on Kubernetes

  apache-airflowkubernetesdockerastronomer-platformastronomer-software
  Language:Python 485

• andreax79 / airflow-code-editor

  A plugin for Apache Airflow that allows you to edit DAGs in browser

  airflowairflow-pluginapache-airflowpython
  Language:Vue 453

• apache / airflow-client-python

  Apache Airflow - OpenApi Client for Python

  airflowapacheapache-airflowapache-airflow-clientpython
  Language:Python 436

• blockchain-etl / ethereum-etl-airflow

  Airflow DAGs for exporting, loading, and parsing the Ethereum blockchain data. How to get any Ethereum smart contract into BigQuery https://towardsdatascience.com/how-to-get-any-ethereum-smart-contract-into-bigquery-in-8-mins-bab5db1fdeee

  apache-airflowblockchain-analyticscryptocryptocurrencydata-analyticsdata-engineeringethereumetlgcpgoogle-cloudgoogle-cloud-platformon-chain-analysisweb3
  Language:Python 427

• astronomer / astro-cli

  CLI that makes it easy to create, test and deploy Airflow DAGs to Astronomer

  apache-airflowastronomer-platformastronomer-softwarekubernetes
  Language:Go 424

• astronomer / astro-sdk

  Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

  pythonsqlpandasairflowsqlitebigquerypostgressnowflakegcss3etleltdagsworkflowsdata-analysisdata-scienceapache-airflow
  Language:Python 375

• airflow-plugins / Example-Airflow-DAGs

  Example DAGs using hooks and operators from Airflow Plugins

  apache-airflowdagetlairflow-pluginsseleniumgoogle-analyticsfacebook-adssftpmarketosalesforcehubspotmarketo-sdkimapairflowmongodbmailgunsingerdbt
  Language:Python 347

• teamclairvoyant / airflow-rest-api-plugin

  A plugin for Apache Airflow that exposes rest end points for the Command Line Interfaces

  airflowrest-apipluginapache-airflowairflow-webserverairflow-plugin
  Language:Python 326

• GoogleCloudPlatform / airflow-operator

  Kubernetes custom controller and CRDs to managing Airflow

  kuberneteskubernetes-operatorapache-airflowairflowcrdkubernetes-controllerworkflow-engineairflow-operator
  Language:Go 299

• astronomer / airflow-chart

  A Helm chart to install Apache Airflow on Kubernetes

  airflowapache-airflowastronomer-softwarehelm-chartkubernetes
  Language:Python 288

• airscholar / e2e-data-engineering

  An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. All components are containerized with Docker for easy deployment and scalability.

  apache-airflowapache-kafkaapache-sparkapache-zookeeperbig-datacassandracontainerizationdata-engineeringdata-pipelinedata-processingdata-storagedockeretl-pipelinepostgresqlreal-time-analytics
  Language:Python 287

• tkp-archive / paperboy

  A web frontend for scheduling Jupyter notebook reports

  jupyterjupyterlabjupyter-notebookspapermillnteractairflowapache-airflowphosphorjsdaskscheduling-notebooksjupyter-notebooknotebookdockerkubernetesluigicelery
  Language:Python 254

• teamclairvoyant / airflow-scheduler-failover-controller

  A process that runs in unison with Apache Airflow to control the Scheduler process to ensure High Availability

  airflowairflow-clusterairflow-schedulerapache-airflowhigh-availabilityscheduler-failover-controller
  Language:Python 235

• kaxil / airflowctl

  A CLI tool to streamline getting started with Apache Airflow™ and managing multiple Airflow projects

  airflowapache-airflowclidagspython
  Language:Python 223

• apache / airflow-client-go

  Apache Airflow - OpenApi Client for Go

  airflowapacheapache-airflowapache-airflow-clientgo
  Language:Go 213

• xnuinside / airflow_in_docker_compose

  Apache Airflow in Docker Compose (for both versions 1.10.* and 2.*)

  docker-compose-filesairflowdocker-airflowcelery-executordockerdocker-composedocker-compose-templateapache-airflow
  Language:Python 186

• rolanddb / airflow-on-kubernetes

  A guide to running Airflow on Kubernetes

  apache-airflowkubernetes
  173

• Toloka / dbt-af

  Distributed run of dbt models using Airflow

  airflowapache-airflowdbtpython
  Language:Python 165

• gestaogovbr / Ro-dou

  Gerador de DAGs no Apache Airflow para fazer clipping do Diário Oficial da União.

  airflow-dagsapache-airflowdados-abertosdiario-oficial-da-uniaogovernment-gazetteopen-data
  Language:Python 164

• airscholar / RedditDataEngineering

  This project provides a comprehensive data pipeline solution to extract, transform, and load (ETL) Reddit data into a Redshift data warehouse. The pipeline leverages a combination of tools and services including Apache Airflow, Celery, PostgreSQL, Amazon S3, AWS Glue, Amazon Athena, and Amazon Redshift.

  apache-airflowawscelerydata-pipelineend-to-end-data-engineeringreddit
  Language:Python 156

• astronomer / astronomer-providers

  Airflow Providers containing Deferrable Operators & Sensors from Astronomer

  airflowairflow-operatorsairflow-providersapache-airflowpythonworkflow
  Language:Python 149

• ninja-van / airflow-boilerplate

  A complete development environment setup for working with Airflow

  airflowpythonapache-airflowpycharmboilerplate
  Language:Python 127

• datacamp / viewflow

  Viewflow is an Airflow-based framework that allows data scientists to create data models without writing Airflow code.

  airflowdata-sciencepackagespythonapache-airflowworkflowdata-engineering
  Language:Python 125

• Wittline / uber-expenses-tracking

  The goal of this project is to track the expenses of Uber Rides and Uber Eats through data Engineering processes using technologies such as Apache Airflow, AWS Redshift and Power BI.

  data-engineeringpythonapache-airflowetl-pipelineaws-redshiftpower-bidata-modelinguber-eatsuber-dataexpenses-dashboardexpenses-trackerairflow-dockeruberaws
  Language:Jupyter Notebook 122

• idealista / airflow-role

  Ansible role to install Apache Airflow

  airflowairflow-roleansibleansible-roleapache-airflowdebian
  Language:YAML 85

• PowerDataHub / terraform-aws-airflow

  Terraform module to deploy an Apache Airflow cluster on AWS, backed by RDS PostgreSQL for metadata, S3 for logs and SQS as message broker with CeleryExecutor

  apache-airflowairflowterraformterraform-modulesawsceleryterraform-modulehacktoberfest
  Language:HCL 84

• caoergou / airflow-extended-api-plugin

  Yet another Airflow plugin using CLI command as RESTful api, supports Airflow v2.X.

  airflow-pluginairflowapache-airflowrestful-apicommand-lineapi
  Language:Python 78