elt

There are 31 repositories under elt topic.

• airflow

  apache / airflow

  Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

  airflowapacheapache-airflowautomationdagdata-engineeringdata-integrationdata-orchestratordata-pipelinesdata-scienceeltetlmachine-learningmlopsorchestrationpythonschedulerworkflowworkflow-engineworkflow-orchestration
  Language:Python 34607

• airbytehq / airbyte

  The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

  datapipelinedata-analysisdata-engineeringjavapythonetlchange-data-capturedata-collectiondata-integrationeltbigqueryredshiftsnowflakedata-pipelinemssqlmysqlpostgresqls3self-hosted
  Language:Python 14153

• apache / doris

  Apache Doris is an easy-to-use, high performance and unified analytics database.

  olapdatabasehadoophivehudiicebergreal-timesqlbigquerydbtdelta-lakeeltetllakehousequery-engineredshiftsnowflakespark
  Language:Java 11413
• dbt-core

  dbt-labs / dbt-core

  dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.

  dbt-viewpointslackpypadata-modelingbusiness-intelligenceanalyticselt
  Language:Python 8931

• apache / seatunnel

  SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.

  data-integrationhigh-performanceofflinereal-timeapachebatchcdcchange-data-capturedata-ingestioneltstreaming
  Language:Java 7399

• mage-ai / mage-ai

  🧙 Build, run, and manage data pipelines for integrating and transforming data.

  machine-learningartificial-intelligencedatadata-engineeringdata-sciencepythoneltetlpipelinesdata-pipelinesorchestrationdata-integrationsqlsparkdbtpipelinereverse-etltransformation
  Language:Python 7086
• kestra

  kestra-io / kestra

  Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.

  workflowworkflow-engineorchestrationschedulerdata-pipelineeltetldatadata-engineeringdata-orchestrationdata-orchestratorlow-codedata-integrationreverse-etldata-qualitypipeline
  Language:Java 6499
• cloudquery

  cloudquery / cloudquery

  The open source high performance ELT framework powered by Apache Arrow

  awsgcpazuresqldata-integrationeltetletl-frameworkbigquerydata-collectiondata-engineeringkubernetesdataairbytegithub-apidata-analysisgooglegocspmattack-surface-management
  Language:Go 5599

• apache / flink-cdc

  Flink CDC is a streaming data integration tool

  change-data-capturecdcbatchdata-integrationdata-pipelinedistributedeltetlflinkkafkamysqlpaimonpostgresqlreal-timeschema-evolution
  Language:Java 5264

• dlt-hub / dlt

  data load tool (dlt) is an open source Python library that makes data loading easy 🛠️

  datadata-engineeringdata-lakedata-loadingdata-warehouseeltextractloadpythontransform
  Language:Python 1751

• meltano / meltano

  Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.

  dataopsdataops-platformeltopen-sourceopensourcedatapipelinesextract-datameltanomeltano-sdkconnectorsintegrationtaptapstargettargetsloadersdata-pipelinessingerdata-engineering
  Language:Python 1602

• TobikoData / sqlmesh

  Efficient data transformation and modeling framework that is backwards compatible with dbt.

  dataengineeringdataopsdbteltetlpythonsqltransformation
  Language:Python 1296

• dataform-co / dataform

  Dataform is a framework for managing SQL based data operations in BigQuery

  data-pipelineseltdata-engineeringbusiness-intelligenceanalyticsetlhacktoberfest
  Language:TypeScript 793
• kuwala

  kuwala-io / kuwala

  Kuwala is the no-code data platform for BI analysts and engineers enabling you to build powerful analytics workflows. We are set out to bring state-of-the-art data engineering tools you love, such as Airbyte, dbt, or Great Expectations together in one intuitive interface built with React Flow. In addition we provide third-party data into data science models and products with a focus on geospatial data. Currently, the following data connectors are available worldwide: a) High-resolution demographics data b) Point of Interests from Open Street Map c) Google Popular Times

  datadata-integrationdata-scienceopen-dataspatial-analysiseltkuwalaopen-sourcescrapingdbtpostgrespysparkpythonjupyteradmin-boundariesgoogle-trendspopulationreactno-codereact-flow
  Language:JavaScript 766

• raystack / optimus

  Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.

  airflowetlworkflowsautomationgolangbigquerydata-warehouseanalyticsdata-modellinganalytics-engineeringdata-transformationdata-pipelineseltbusiness-intelligencedataops
  Language:Go 737
• quary

  quarylabs / quary

  BI for engineers

  analyticsbusiness-intelligencedata-modelingelt
  Language:TypeScript 579

• artie-labs / transfer

  Database replication platform that leverages change data capture. Stream production data from databases to your data warehouse (Snowflake, BigQuery, Redshift) in real-time.

  snowflakecdcchange-data-capturegolangbigquerykafkaapache-kafkadata-integrationdata-pipelinesdatabasedebeziumeltredshift
  Language:Go 533
• automate-dv

  Datavault-UK / automate-dv

  A free to use dbt package for creating and loading Data Vault 2.0 compliant Data Warehouses (powered by dbt, an open source data engineering tool, registered trademark of dbt Labs)

  datavault20dbtdatawarehousingdatavaultetlsnowflakedataengineeringdatawarehousedatalakeeltdata-vaultsqlmetadata
  459

• gouline / dbt-metabase

  dbt + Metabase integration

  dbtmetabasepythonanalyticsbusiness-intelligencedatadata-modellingeltpypavizualisation
  Language:Python 430
• versatile-data-kit

  vmware / versatile-data-kit

  One framework to develop, deploy and operate data workflows with Python and SQL.

  data-sciencedata-engineeringsqltrinodata-lineageetleltdata-pipelinesdata-engineerdata-warehousewarehouseanalyticssnowflakedataopsdata-engineering-pipelinepythondatapipelinedata-structuresdatabase
  Language:Python 410

• osalvador / ReplicaDB

  ReplicaDB is open source tool for database replication, designed for efficiently transferring bulk data between relational and non-relational databases

  cross-platformdatabasedatabase-replicationeltetlfastjavakafkamysqlnosqlnosql-databasesoraclepostgresqlsqltransactional
  Language:Java 362

• aws-samples / aws-etl-orchestrator

  A serverless architecture for orchestrating ETL jobs in arbitrarily-complex workflows using AWS Step Functions and AWS Lambda.

  awsglueetleltstepfunctionsorchestrationlambdaserverlessbig-databigdataextracttransformloadathenaamazon-web-servicesstate-machine
  Language:Python 323

• astronomer / astro-sdk

  Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

  pythonsqlpandasairflowsqlitebigquerypostgressnowflakegcss3etleltdagsworkflowsdata-analysisdata-scienceapache-airflow
  Language:Python 320

• cuebook / cuelake

  Use SQL to build ELT pipelines on a data lakehouse.

  apache-icebergdeltalakehousedatalakedata-lakeeltetldata-engineeringdata-integrationdata-ingestionapache-sparkspark-sqlupsertincremental-updatesdata-transferpipelinesdata-pipelinezeppelin-notebooksql
  Language:JavaScript 283
• sling-cli

  slingdata-io / sling-cli

  Sling is a CLI tool that extracts data from a source storage/database and loads it in a target storage/database.

  extractloadetlelt
  Language:Go 263

• datacoves / dbt-coves

  CLI tool for dbt users to simplify creation of staging models (yml and sql) files

  analyticsbigquerydatacovesdbteltetljinjapythonredshiftsnowflakesql
  Language:Python 209

• umitkaanusta / reddit-detective

  Play detective on Reddit: Discover political disinformation campaigns, secret influencers and more

  analyticsdatasocial-networksocial-mediaredditapietleltanalysisnetworksocialpoliticsgraphdatabasegraph-databaseneo4j
  Language:Python 206

• DataRecce / recce

  Data Reconnaissance - pull request review tool for dbt projects

  codereviewdbtelt
  Language:TypeScript 168

• airbytehq / PyAirbyte

  PyAirbyte brings the power of Airbyte to every Python developer.

  data-engineeringeltpython
  Language:Python 140
• airbyte_serverless

  unytics / airbyte_serverless

  Airbyte made simple (no UI, no database, no cluster)

  airbytebigquerydatadata-analysisdata-engineeringdata-warehouseeltetlpipeline
  Language:Python 119
• sayn

  173TECH / sayn

  Data processing and modelling framework for automating tasks (incl. Python & SQL transformations).

  analyticsetldata-modelingdata-engineeringdata-sciencepythonsqlautomationeltdata-extraction
  Language:Python 117

• faros-ai / airbyte-connectors

  Airbyte connectors (sources & destinations) + Airbyte CDK for JavaScript/TypeScript

  npmairbytecdktypescriptjavascriptfarosconnectorsairbyte-connectorsetleltcicdsdlcairbyte-sourcesairbyte-destinationsairbyte-cdk
  Language:TypeScript 102

• yokawasa / databricks-notebooks

  Collection of Sample Databricks Spark Notebooks ( mostly for Azure Databricks )

  databricksazureazuredatabrickspythonsparkeltstreaming
  Language:Jupyter Notebook 76

• codeforkjeff / dbt-sqlite

  A SQLite adapter plugin for dbt (data build tool)

  dbtsqliteeltetl
  Language:Python 73
• dbd

  zsvoboda / dbd

  dbd is a database prototyping tool that enables data analysts and engineers to quickly load and transform data in SQL databases.

  eltsqldatabasedatabase-schemasetlcsvjsonparquetxlsxxlsexcelpythonpython3postgresqlmysqlsqlitesnowflakebigqueryredshift
  Language:Python 56
• getl

  ascrus / getl

  A tool for developing and testing ETL and ELT processes for automating the capture, delivery and processing of information in data warehouses on the MicroFocus Vertica platform.

  verticaetleltunit-testingdslcsvkafkaexcelsqljsonxmlhdfshiveimpala
  Language:Groovy 54