elt-pipeline

There are 0 repository under elt-pipeline topic.

• Fozan-Talat / divvy-bikeshare-de-project

  An end-to-end data pipeline which extracts divvy bikeshare data from web loads it into data lake and datawarehouse transforms it using dbt and finally , a dashboard to visualize the data using looker studio, the pipeline is orchestrated using prefect

  bigquerydbtdivvy-bikesgcpgcs-bucketgoogle-cloud-platformgoogle-cloud-storagelooker-studioprefectpythonterrafromdata-engineeringdata-modelingdata-pipelinedata-visualizationdata-warehouseelt-pipelineorchestrationdezoomcamp
  Language:Python 34

• dataforgelabs / dataforge-core

  DataForge helps data teams write functional transformation pipelines by leveraging software engineering principles

  analyticsdata-modelingeltelt-pipelineetletl-framework
  Language:PLpgSQL 25

• Mg30 / pydwt

  Modeling tool like DBT to use SQL Alchemy core with a DataFrame interface like

  dbtpython3sqlsqlalchemy-pythondatabasedatapipelineelt-pipeline
  Language:Python 10

• bennyaustin / synapse-dataplatform

  A modern data platform implemented on Azure Synapse Analytics using ELT Framework - https://github.com/bennyaustin/elt-framework. Data platform infrastructure provisioned using https://github.com/bennyaustin/iac-synapse-dataplatform

  pysparkmssparkutilsazure-data-factoryazure-databricksazure-devops-pipelinesazure-synapse-analyticsazure-synapse-pipelinesazure-synapse-serverless-sqlelt-frameworkelt-pipelinegithub-actions
  Language:TSQL 6

• jgrove90 / ufo-deltalake

  🛸 This project showcases an Extract, Load, Transform (ELT) pipeline built with Python, Apache Spark, Delta Lake, and Docker. The objective of the project is to scrape UFO sighting data from NUFORC and process it through the Medallion architecture to create a star schema in the Gold layer that is ready for analysis.

  apache-sparkdata-engineeringdata-pipelinedelta-lakedockerelt-pipelinepython
  Language:Python 3

• stellar / stellar-dbt-public

  Public DBT instance to aid in data transformation for analytics purposes

  elt-frameworkelt-pipelinestellarstellar-data-movementstellar-etl
  Language:Shell 3

• arunp77 / SQL

  SQL, Databases, warehouses, Data lake, cloud storage, MYSQL, Data Pipeline

  airflow-dagsaws-ec2aws-s3data-pipelinedatabase-managementdatawarehouseelt-pipelineetl-pipelinemysql-databasesnowflakesqlapache-airflowapache-nifisnowflake-schemasnowflakessnowpipe
  2

• Baitur5 / reddit_api_elt

  airflowdbteltelt-pipelineetletl-pipelinereddit-apisubreddit-analyticssubreddit-data
  Language:Python 2

• jackmulligan-ire / ppr-pipeline

  Irish Property Price Register transformed into a data warehouse via an EtLT pipeline.

  dockerpythons3awsboto3data-warehouseelt-pipelineetl-pipelinefact-tabledata-engineeringdata-engineering-pipeline
  Language:TypeScript 2

• jgrove90 / rick-and-morty-deltalake

  🍺 A data engineering project showcasing an ELT pipeline using modern technologies such as Delta-rs, and Apache Airflow.

  apache-airflowdata-engineeringdata-pipelinedelta-lakedelta-tablesdockerpandaspythonelt-pipeline
  Language:Python 2

• Suprame4 / Data_Engineering_Projects

  Data engineering projects

  airflowdata-modelingelt-pipelineetl-pipelineibmdb2mysqlpandaspythonpythoonsqletlkafkamysql-databasecassandracloudantmongodbbash-script
  Language:Jupyter Notebook 2

• JaleesMoeen / Crowdfunding_ETL

  Builded an ETL pipeline using Python, Pandas, Python dictionary methods and regular expressions to ETL data. It involves extracting data from multiple sources, cleaning and transforming the data using Jupyter Notebook with pandas, numpy, and datetime packages, and loading the cleaned data into a relational database using pgAdmin

  elt-pipelinejupyter-notebookpython
  Language:Jupyter Notebook 1

• johnkdunyo / Postgres-ELT-Data-Pipeline

  An ELT data pipeline project utilizing docker and postgress (both source and destination dbs)

  dockerelt-pipelinepostgresql
  Language:Python 1

• MettaSurendhar / DataEngineeringProject

  Data Engineering project which involves ETL using PostgreSQL and Python

  apidataengineeringdataengineering-adataengineering-mdataengineering-wdockerdocker-containerdocker-imagedockerfileeltelt-pipelineextractionloadmovie-databasemovies-apipostgrespostgresqlpythonpython3tranformations
  Language:Python 1

• ArchitHallan / Crowdfunding-Project

  ELT Project

  eltelt-pipeline
  Language:Jupyter Notebook 0

• dondogecl / cool_data_pipeline

  Data pipeline from RDBMS to AWS

  aws-redshiftaws-s3data-engineeringelt-pipelinerdbms
  Language:Python 0

• hdt94 / dtc-de-project

  Data engineering project for TLC taxi Parquet data following an ELT model (extraction, load, transform)

  airflowbigquerydata-engineeringdbtdockereltelt-pipelinegcpgoogle-cloud-platformorchestrationterraform
  Language:Python 0

• jolares / demo-dbt

  Example ELT data pipeline project using dbt

  dbtelt-pipelinegithub-actionssnowflake
  Language:Shell 0

• judeleonard / e-commerce_activity_tracking

  This is an ELT data pipeline setup to track the activities of an e-commerce website based on orders, reviews, deliveries and shipment date. This project utilized technologies like Airflow, AWS RDS-Postgres, Python etc.

  airflowairflow-dagairflow-dockerdataengineeringdatapipelinee-commerceelt-pipelinepostgres
  Language:Python 0

• kayazay / zomato_analytics

  An end to end ELT project that uses data from the Zomato Restaurant, an Indian multinational restaurant aggregator and food delivery company. The project extracts data from Kaggle dataset, loads it into Snowflake tables, then is transformed and modelled in dbt Labs.

  business-intelligencedata-engineeringdata-visualizationdata-warehousedbtelt-pipelinekaggle-datasetsnowflakesqltableauzomato
  0

• seilylook / custom-elt-project

  airflowdataengdbtdockereltelt-pipelinepythonscratch
  Language:Python 0

• shahinyusifli / dw-credit-risk

  A system has been set up for analyzing credit risk, involving a data warehouse and pipeline. The tools used for this solution include Prefect for workflow management, Redshift for the data warehouse, and an S3 bucket for storage.

  batch-processingdatawarehouseelt-pipelineprefectredshift
  Language:Python 0

• vishu-tyagi / BigQuery-ELT

  BigQuery data pipeline with dbt, Spark, Docker, Airflow, Terraform, GCP

  airflowbatch-processingbig-data-analyticsbig-data-processingbigquerydbtdockereltelt-pipelineetl-pipelinepysparkpythonsparkterraform
  Language:Python 0

• andressagomes26 / adventure_works_analytics

  Neste projeto, são realizadas as transformações dos dados brutos da empresa Adventure Works (AW). 🚴‍♀️

  analytics-engineeringbigquerydata-analyticsdbtelt-pipelinesql
  Language:Jupyter Notebook

• ClaytonAllenThompsonII / WebApp

  Django Web Application for inventory and invoice management.

  aws-dynamodbaws-lambdaaws-s3django-applicationdjango-channelsec2-instanceelt-pipelinepostgresvpcredis
  Language:Python

• ocittt / ELT_project-w-DBT

  ELT Project that ultilizes Docker and PostgreSQL for a simple ELT process

  dbtdockereltelt-pipelinepostgresqlsql
  Language:Python

• teddyk251 / traffic-data-ELT-pipeline

  An ELT pipeline built for the pNEUMA open dataset of naturalistic trajectories of half a million vehicles collected by a swarm of drones in a congested downtown area of Athens, Greece.

  airflowdbtdockerelt-pipelinepostgresqlredash
  Language:Python