datapipeline
There are 6 repositories under datapipeline topic.
zhaoyachao / zdh_web
大数据采集,抽取平台,zdh_web是zdh系列服务的可视化管理平台，包含数据采集,调度,权限,审批流,私域营销等模块
Language:Java 418- Data-Engineering-Roadmap
ErdemOzgen / Data-Engineering-Roadmap
Roadmap for Data Engineering
Language:Java 148 - TensorPipe
kartik4949 / TensorPipe
High Performance Tensorflow Data Pipeline with State of Art Augmentations and low level optimizations.
Language:Python 86 Alireza-Akhavan / tf2-tutorial
Tensorflow 2 Tutorials (use tensorflow and keras in a better way!)
Language:Jupyter Notebook 51cloudposse / terraform-aws-efs-backup
Terraform module designed to easily backup EFS filesystems to S3 using DataPipeline
Language:HCL 42- Language:Scala 28
kromozome2003 / Snowflake-Json-DataPipeline
Building Json data pipeline within Snowflake using Streams and Tasks
Language:TSQL 27- kedro-static-viz
WaylonWalker / kedro-static-viz
kedro cli plugin for generating a static kedro viz site (html, css, js) that can be deployed on many serverless tools.
Language:Python 27 covalenthq / bsp-geth
Ethereum client written in Go, modified for full-hierarchy data exports and block specimen production
Language:Go 26shazam / scala-datapipeline-dsl
Domain-specific language to help build and maintain AWS Data Pipelines
Language:Scala 26KennethanCeyer / awesome-data-pipeline
Awesome list for datapipeline
- kedro-action
A GitHub Action to lint, test, build-docs, package, and run your kedro pipelines. Supports any Python version you'll give it (that is also supported by pyenv).
Language:Shell 20 NVIDIA / go-tfdata
Go library that provides easy-to-use interfaces and tools for TensorFlow users, in particular allowing to train existing TF models on .tar and .tgz datasets
Language:Go 14DudeWhoCode / kulay
High speed message passing between various queues and services
Language:Go 12adilkhash / luigi-course-materials
Материалы для курса Введение в Data Engineering: дата пайплайны
Language:Python 11- Language:Python 10
wri / gfw_forest_loss_geotrellis
Global Tree Cover Loss Analysis using Geotrellis and SPARK
Language:Scala 10multilayer-io / airflow-kubernetes
Simple Airflow on Kubernetes (GKE)
Language:Python 9- Language:Python 9
huanwuji / teleporter
Reactive Streams distributed datapipeline for data process. Now support kafka,jdbc,kudu,elasticsearch,hdfs.etc
Language:Scala 8- GTFS-Data-Pipeline-TfNSW-Bus
teckkean / GTFS-Data-Pipeline-TfNSW-Bus
GTFS Data Pipeline for TfNSW Bus Datasets
Language:Jupyter Notebook 7 - Language:Python 7
Yan-Luo-AU / Data_Engineer_Project_ETL_BI
This is an ETL project - extracting data from an ecommerce transactional database on RDS, transforming the data using AWS glue job, and loading it to a Redshift data warehouse, and connected it to Tableau for BI
Language:Python 7AviatorIfeanyi / etl_with_mage_ai
An ETL data pipeline that extracts data from source and loads it to destination, automated using mage.ai
Language:Python 5mehroosali / databricks-F1-Project
A data pipeline project build on databricks and azure to demostrate lifecycle of a cloud data project.
Language:Jupyter Notebook 5- Language:Python 5
ankitanshumanmohapatra / Azure-Olympics-Analysis-Data-Engineering-End-to-End-Project
This is a End-to-End Azure Data Engineering Project | Analysis on the entire ETL Pipeline - Azure Factory, Azure Lake Gen 2, Databricks, Azure Synapse Analytics & Dashboards
Language:Jupyter Notebook 4gchatterjee-git / Data-Pipeline-AWS
This is a project which demonstrates creation of a data pipeline by scraping data using twitter API and creating a data delivery stream using Kinesis Firehose for ingesting data to Amazon S3.
Language:Python 4Kushalkhadka7 / dagster_clickhouse_dbt
DBT and clickhouse test project with dagster
Language:Python 4- Language:Python 4
RudraChatterjee / Machine-Failure_Prediction_EnsembleMethods_ModelTuning
This project predicts wind turbine failure using numerous sensor data by applying classification based ML models that improves prediction by tuning model hyperparameters and addressing class imbalance through over and under sampling data. Final model is productionized using a data pipeline
Language:Jupyter Notebook 4Sagar-Salvi / Data-Engineering-Project
The Centralized Data Warehouse and ML Solution for Banking Analytics is a project that combines a centralized repository for banking data with machine learning algorithms to enable predictive analysis.
Language:Jupyter Notebook 4aduo122 / Cryptocurrency-price-analysis-platform
A data pipeline to analyze real time cryptocurrency price
Language:Python 3ekhtiar / swiss-transport-datapipeline
A data pipeline to daily pull public transport data from the opentransportdata.swiss portal. This pipeline has three tasks, pull the right data from opentransportdata.swiss, push the data to s3 for storage, and transform and load the transformed data to a database. Hopefully this repository helps people explain ETL / Batch data pipeline.
Language:Python 3sanketsanap5 / EScooter-Data-Pipeline
An Automated data pipeline using "Apache Airflow" performing "ETL" on RAW data using "Pandas" library then stage data into "PostgreSQL" then process it distributed cluster and parallelly using "Spark" and loaded final useful data into "ElasticSearch" NoSQL DB warehouse
Language:Python 3- Language:Python 3
