etl-pipeline

There are 38 repositories under etl-pipeline topic.

• orchest / orchest

  Build data pipelines, the easy way 🛠️

  data-sciencemachine-learningpipelinesidejupytercloudself-hostedjupyterlabnotebooksdockerpythondata-pipelinesorchestdeploymentkubernetesairflowdagetletl-pipeline
  Language:TypeScript 4020
• incubator-streampark

  apache / incubator-streampark

  Make stream processing easier! Easy-to-use streaming application development framework and operation platform.

  streamingstreamparkapachedevelopment-frameworkeasy-to-useetl-pipelineoperation-platform
  Language:Java 3694

• AlexIoannides / pyspark-example-project

  Implementing best practices for PySpark ETL jobs and applications.

  data-engineeringdata-scienceetletl-jobetl-pipelinepysparkpythonspark
  Language:Python 1381
• Udacity-Data-Engineering-Projects

  san089 / Udacity-Data-Engineering-Projects

  Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.

  datadata-engineeringdata-engineering-pipelineetl-pipelinecassandra-databasepostgresql-databasedata-modelingdata-warehousedata-lakeairflowairflow-operatorsclustercassandrainfrastructurepostgresawsaws-ec2aws-sdkaws-s3cloudformation
  Language:Python 1341

• DAGWorks-Inc / hamilton

  Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage and metadata. Runs and scales everywhere python does.

  data-sciencepythondagdata-engineeringdataframeetletl-frameworketl-pipelinefeature-engineeringfeaturizationmachine-learningnumpypandassoftware-engineeringdata-analysislineagellmopsmlopsorchestrationhacktoberfest
  Language:Jupyter Notebook 1308
• goodreads_etl_pipeline

  san089 / goodreads_etl_pipeline

  An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

  etl-pipelineetl-frameworksparkapache-sparkapache-airflowairflowredshiftemr-clusterlivys3warehousedata-lakeschedulerdata-migrationdata-engineeringdata-engineering-pipelinepythongoodreads-data-pipelineairflow-dagetl-job
  Language:Python 1219

• stitchfix / hamilton

  A scalable general purpose micro-framework for defining dataflows. THIS REPOSITORY HAS BEEN MOVED TO www.github.com/dagworks-inc/hamilton

  pythonpandasdagdata-sciencedata-engineeringnumpysoftware-engineeringetl-frameworketl-pipelineetlfeature-engineeringfeaturizationdataframestitch-fixdata-platformhamiltonhamiltonianmachine-learning
  Language:Python 874

• techascent / tech.ml.dataset

  A Clojure high performance data processing system

  clojuredataframecsvxlsxdatasciencemachine-learningdatasetetl-pipelinejava
  Language:Clojure 631

• YotpoLtd / metorikku

  A simplified, lightweight ETL Framework based on Apache Spark

  big-datasparkscalaetl-frameworkdistributed-computingsqletletl-pipeline
  Language:Scala 576
• flow

  flow-php / flow

  Flow PHP - data processing framework

  etletl-pipelineetl-framework
  Language:PHP 348
• setl

  SETL-Framework / setl

  A simple Spark-powered ETL framework that just works 🍺

  sparketlframeworkscalasetlpipelinedata-transformationdata-sciencedata-engineeringdata-analysismodularizationdatasetbig-dataetl-pipelinemachine-learning
  Language:Scala 176

• realize-engineering / pipebird

  Pipebird is open source infrastructure for securely sharing data with customers.

  data-infrastructuredata-pipelinesdata-sharingopen-sourcedata-engineeringetl-pipelineself-hosteddata-pipeline
  Language:TypeScript 167

• Indexical-Metrics-Measure-Advisory / watchmen-matryoshka-doll

  Watchmen Platform is a low code data platform for data pipeline, meta data management , analysis, and quality management

  visualizationchartsdata-visualizationdata-pipelineetl-pipelinedata-quality-monitoringpipeline
  Language:Python 131

• data-engineering-community / data-engineering-project-template

  This is a template you can use for your next data engineering portfolio project.

  datadata-engineeringdata-warehouseetletl-pipelinepythonsql
  117

• jitsucom / bulker

  Service for bulk-loading data to databases with automatic schema management (Redshift, Snowflake, BigQuery, ClickHouse, Postgres, MySQL)

  data-engineeringdatawarehouseetletl-pipelineingestionpipeline
  Language:Go 115

• patterns-app / patterns-devkit

  Data pipelines from re-usable components

  data-analysisdata-engineeringdata-pipelinedata-pipelinesdata-scienceetletl-frameworketl-pipelineetl-pipelinesfunctional-reactive-programmingimmutabilitypipelinessql
  Language:Python 106

• usc-isi-i2 / dig-etl-engine

  Download DIG to run on your laptop or server.

  crawlingetl-frameworketl-pipelineinformation-extractioninformation-visualizationsearch-engine
  99

• Wittline / uber-expenses-tracking

  The goal of this project is to track the expenses of Uber Rides and Uber Eats through data Engineering processes using technologies such as Apache Airflow, AWS Redshift and Power BI.

  data-engineeringpythonapache-airflowetl-pipelineaws-redshiftpower-bidata-modelinguber-eatsuber-dataexpenses-dashboardexpenses-trackerairflow-dockeruberaws
  Language:Jupyter Notebook 97

• imsanjoykb / Data-Science-Regular-Bootcamp

  Regular practice on Data Science, Machien Learning, Deep Learning, Solving ML Project problem, Analytical Issue. Regular boost up my knowledge. The goal is to help learner with learning resource on Data Science filed.

  data-sciencemachine-learningartificial-intelligencedeep-learningneural-networkdata-science-projectsdata-visualizationdata-analysisdata-science-notebookdatabase-connectionmysql-databasepostgresqlsqliteetl-pipelineetl-processpython-automationnumpypandaspythonfeature-engineering
  Language:Jupyter Notebook 94

• restarone / violet_rails

  an app engine for your business. Seamlessly implement business logic with a powerful API. Out of the box CMS, blog, forum and email functionality. Developer friendly & easily extendable for your next SaaS/XaaS project. Built with Rails 6, Devise, Sidekiq & PostgreSQL

  rubyonrailsxaassaastemplatemultitenancyviolet-railsmulti-tenancyforumblogsaas-boilerplatecmsruby-on-railsrailsrubywordpress-replacementemberemberjsetl-automationetl-frameworketl-pipeline
  Language:Ruby 93

• airscholar / e2e-data-engineering

  An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. All components are containerized with Docker for easy deployment and scalability.

  apache-airflowapache-kafkaapache-sparkapache-zookeeperbig-datacassandracontainerizationdata-engineeringdata-pipelinedata-processingdata-storagedockeretl-pipelinepostgresqlreal-time-analytics
  Language:Python 86
• ApacheSpark

  martandsingh / ApacheSpark

  This repository will help you to learn about databricks concept with the help of examples. It will include all the important topics which we need in our real life experience as a data engineer. We will be using pyspark & sparksql for the development. At the end of the course we also cover few case studies.

  apachesparkdata-analysisdata-engineeringdatabasedatabricksdatalakedeltalakeetl-pipelinehadoophivesparkspark-sqlspark-streamingtimetraveletlpysparksql
  Language:Python 81

• runprism / prism

  Prism is the easiest way to develop, orchestrate, and execute data pipelines in Python.

  datadata-engineeringdata-scienceetletl-pipelinemachine-learningorchestrationpipelinepythondata-orcbigquerydata-analysisdata-integrationdbtpostgresredshiftsnowflaketrino
  Language:Python 79
• unstract

  Zipstack / unstract

  No-code LLM Platform to launch APIs and ETL Pipelines to structure unstructured documents

  etl-pipelinellm-platformunstructured-data
  Language:Python 79

• sundios / SEO-Dashboard

  SEO dashboard from Search console Data using the Google Search API, Mysql database , NodeJS RESTAPI( ExpressJS) and reactJs Dashboard

  seoseotoolsetl-pipelinedashboardreactseo-monitorrest-apigoogle-search-console-pythongoogle-search-consolemysqlnode-jsexpressjsetl-kpi
  Language:JavaScript 77

• jvalue / jayvee

  Jayvee is a domain-specific language and runtime for automated processing of data pipelines

  data-engineeringdata-pipelinedata-sciencedomain-specific-languageetl-pipelinetypescript
  Language:TypeScript 72

• maxim2266 / csvplus

  csvplus extends the standard Go encoding/csv package with fluent interface, lazy stream operations, indices and joins.

  gocsvcsv-formatfluent-interfacestream-processinggo-csvetl-pipelineetl-frameworketl
  Language:Go 66

• cyber-drop / ethereum_analytical_db

  Ethereum Analytical Database - Ethereum data access solution that can be used for analytics and application development. The solution works on a fast DB - Clickhouse.

  ethereumblockchainetl-pipelineetlclickhouseetherc20erc721erc223dexapiethereum-etl
  Language:HTML 63
• AzureDataFactoryHOL

  Mmodarre / AzureDataFactoryHOL

  Azure Data Factory Hands On Lab - Step by Step - A Comprehensive Azure Data Factory and Mapping Data Flow step by step tutorial

  azure-data-factoryazureadf-pipelinefilter-activitylookup-activityforeach-activitymetadata-activitymapping-dataflowsazure-key-vaultetl-pipelinehands-on-labhands-on-azure-data-factoryazure-data-factory-tutorialazure-modern-data-warehousweb-activityforeach-loop-activity
  63

• mycelial / mycelial

  Move your data with ease.

  data-pipelinesedge-computingetletl-pipelinerust
  Language:Rust 61
• onetl

  MobileTeleSystems / onetl

  One ETL tool to rule them all

  sparketletl-componentsetl-pipelinehwmplugin-systempydantic
  Language:Python 58

• MohamedHmini / tweetsOLAPing

  implementing an end-to-end tweets ETL/Analysis pipeline.

  datawarehousingdatawarehouseetl-pipelinetweetstweets-classificationtweets-scrapertwitter-apigoogle-api-clientapi-clientweb-crawlinganalysiscube-analysismulti-dimensional-analysisssisssas-multidimensionalmultithreadingpowerbi-report
  Language:Python 57
• jira-database-etl

  toddbirchard / jira-database-etl

  :mens: 💾 Script to import issues from a JIRA instance into a database.

  jira-rest-apipython3pandasflaskflask-sqlalchemyetletl-pipeline
  Language:Python 53

• vim89 / datapipelines-essentials-python

  Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations

  apache-sparksparkspark-sqlpythonpython3pysparketletl-pipelineetl-frameworketl-componentsxmlxml-parsingdatalakebig-datahadoophadoop-mapreducehadoop-hdfsdata-pipeline
  Language:Python 53

• josephmachado / bitcoinMonitor

  Near real time ETL to populate a dashboard.

  python3etl-pipelinemetabasecronnearrealtimepostgresdockerdocker-composepytest
  Language:Python 52
• etl-framework

  MassStreetAnalytics / etl-framework

  A framework for moving data into a data warehouse.

  etlpythonsqlserversqletl-frameworketl-pipelineetl-componentsdata-warehouse
  Language:Jupyter Notebook 51