aws-redshift

There are 7 repositories under aws-redshift topic.

• alanchn31 / Data-Engineering-Projects

  Personal Data Engineering Projects

  data-lakeingest-datadata-engineeringdata-warehousecassandraaws-redshiftmongodbscrapysparkairflowpostgresdata-engineering-nanodegreestar-schemadata-modeling
  Language:Jupyter Notebook 961

• tokern / piicatcher

  Scan databases and data warehouses for PII data. Tag tables and columns in data catalogs like Amundsen and Datahub

  aws-athenaaws-glueaws-redshiftcatalogdatadata-catalogdatabasephipiipythonsnowflake
  Language:Python 322

• aws / amazon-redshift-python-driver

  Redshift Python Connector. It supports Python Database API Specification v2.0.

  aws-redshiftdata-sciencedata-analysisamazon-redshift
  Language:Python 216

• alanchn31 / Movalytics-Data-Warehouse

  Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow

  dockerairflowsparksqlpython3redshiftdata-engineering-pipelinedata-engineer-nanodegreepysparkdata-engineeringmovie-databaseaws-s3aws-redshiftanalyticsdata-warehouse-clouddata-modellingmovie-recommendationmovie-reviewsudacity
  Language:Python 157

• Wittline / uber-expenses-tracking

  The goal of this project is to track the expenses of Uber Rides and Uber Eats through data Engineering processes using technologies such as Apache Airflow, AWS Redshift and Power BI.

  data-engineeringpythonapache-airflowetl-pipelineaws-redshiftpower-bidata-modelinguber-eatsuber-dataexpenses-dashboardexpenses-trackerairflow-dockeruberaws
  Language:Jupyter Notebook 122

• shravan-kuchkula / udacity-data-eng-proj-1

  Developed a data pipeline to automate data warehouse ETL by building custom airflow operators that handle the extraction, transformation, validation and loading of data from S3 -> Redshift -> S3

  airflowaws-redshiftdata-pipelinesdocker
  Language:Python 90

• aws-solutions / clickstream-analytics-on-aws

  Clickstream Analytics on AWS source code

  awsaws-amplifyaws-cdkaws-emr-serverlessaws-kinesis-streamaws-mskaws-quicksightaws-redshiftclickstreamdata-analysisaws-clickstream-solutionaws-solutionsweb-analysisweb-analytics
  Language:TypeScript 84

• KentHsu / Udacity-Data-Engineering-Nanodgree

  Udacity Data Engineering Nanodegree Program

  data-engineeringpostgresqlapache-cassandraaws-redshiftaws-s3apache-sparkapache-airflowdata-pipelinesdata-warehousesdata-lakedata-quality
  Language:Jupyter Notebook 52

• jackmleitch / StravaDataPipline

  :arrows_counterclockwise: :running: EtLT of my own Strava data using the Strava API, MySQL, Python, S3, Redshift, and Airflow

  pythonawsaws-s3aws-redshiftdata-engineeringpostgresqlsqlairflow
  Language:Python 32

• AnMol12499 / Reddit-Analytics-Integration-Platform

  Project was based on an interest in Data Engineering, ETL pipeline. It also provided a good opportunity to develop skills and experience in a range of tools. As such, project is more complex than required, utilising dbt, airflow, docker and cloud based storage.

  airflowaws-redshiftaws-s3dbtdockeretl-pipelinegoogle-studiopythonterraform
  Language:Python 30

• heroku-examples / analytics-with-kafka-redshift-metabase

  An example system that captures a large stream of product usage data, or events, and provides both real-time data visualization and SQL-based data analytics.

  aws-redshiftmetabasekafkaherokudata-analyticsdata-visualization
  Language:JavaScript 26
• aws-data-pipeline

  ismaildawoodjee / aws-data-pipeline

  A batch processing data pipeline, using AWS resources (S3, EMR, Redshift, EC2, IAM), provisioned via Terraform, and orchestrated from locally hosted Airflow containers. The end product is a Superset dashboard and a Postgres database, hosted on an EC2 instance at this address (powered down):

  pythondockerpostgresqlairfloweltetldata-engineeringawsaws-s3aws-emraws-redshiftaws-ec2aws-iamterraformapache-supersetinfrastructure-as-codedata-pipelinesql
  Language:Python 23

• moritzkoerber / covid-19-data-engineering-pipeline

  A Covid-19 data pipeline on AWS featuring PySpark/Glue, Docker, Great Expectations, Airflow, and Redshift, templated in CloudFormation and CDK, deployable via Github Actions.

  awsaws-ecraws-glueaws-lambdaaws-s3dockergreat-expectationspysparksparkapiaws-cdkaws-redshiftaws-cloudformationapache-airflowapache-spark
  Language:Python 23

• LoveNui / DataEngineering-Capstone-Project

  airflowaws-redshiftaws-s3data-engineeringpythonsparksql
  Language:Jupyter Notebook 16

• tmheo / spring-data-jpa-redshift-sample

  spring boot data jpa integration with aws redshift sample

  spring-bootspring-data-jpaaws-redshift
  Language:Java 15

• kishlayjeet / Zomato-Twitter-Sentiment-Analysis-Data-Pipeline

  This project provides valuable customer sentiment insights for Zomato by tracking and analyzing tweets related to their brand and services.

  airflowaws-lambdaaws-redshiftaws-s3boto3data-engineeringdata-pipelineetlnltkpandaspsycopg2pythonseleniumtwitter-data-pipelinetwitter-sentiment-analysiszomato-data-analysissentiment-data-pipelinevedar-lexiconzomato-data-pipeline
  Language:Python 14

• AWS-Big-Data-Projects / Analysing-Census-Data-using-aws

  Use aws-emr and aws-redshift to analyse dataset of adult census of USA

  aws-emraws-redshiftapache-sparkaws-s3
  13

• vsouza / spark-kinesis-redshift

  Example project for consuming AWS Kinesis streamming and save data on Amazon Redshift using Apache Spark

  spark-streamingsparkawsaws-kinesisaws-kinesis-streamaws-redshiftpythonshelletletl-pipeline
  Language:Python 11

• kishaningithub / rdapp

  rdapp - Redshift Data API Postgres Proxy

  aws-redshifthacktoberfestredshiftawsgogolang
  Language:Go 10

• lenguyenthedat / aws-redshift-to-rds

  A simple command-line tool to copy tables from Amazon Redshift to Amazon RDS (PostgreSQL).

  aws-redshiftawsamazon-redshiftrdsamazon-rdshaskell
  Language:Haskell 10

• essraahmed / Data-Warehouse-With-Redshift

  Data Warehouse with AWS Redshift and Visualizing data using Power BI

  awsaws-redshiftclouddatadata-engineeringdatawarehousedatawarehousingdwhetletl-pipelineredshiftredshift-clusterredshift-databasepowerbianalyticssql
  Language:Jupyter Notebook 8

• taise / Spectrometer

  AWS Redshift monitoring web console

  aws-redshiftmonitoringruby
  Language:Slim 7

• twistedFantasy / aws

  The goal of this repository is to provide good and clear examples of Amazon CLI commands together with Amazon CDK to easily create any AWS services and resources

  amazon-awsamazon-cdkamazon-web-servicesaws-codedeployaws-ec2aws-elasticsearchaws-iamaws-load-balanceraws-parameter-storeaws-rdsaws-redshiftaws-route53aws-s3aws-security-groupaws-sqsaws-systemmanageraws-vpcpythonpython3
  Language:Python 7

• aws-samples / zero-etl-architecture-patterns

  Zero-ETL integrations - Enable near real-time analytics on petabytes of transactional data

  aws-aurora-databaseaws-redshiftzero-etl
  Language:Python 6

• DivineSamOfficial / SmartCityProject

  Smart City Realtime Data Engineering Project

  awsaws-athenaaws-ec2aws-quicksightaws-redshiftaws-s3kafkapysparkpythonspark-streamingaws-glueaws-glue-crawleraws-glue-data-catalog
  Language:Python 5

• polo2444172276 / Udacity-Data-Engineering-Nanodegree

  Completed Udacity's data engineering nano degree. Went through a series of exercises and projects to learn and practice the trendy big data management tools.

  airflowawsaws-ec2aws-redshiftaws-s3cassandradata-engineeringdata-lakedata-warehouseetl-pipelinepostgresqlspark
  Language:PLpgSQL 5

• FedericoSerini / DEND-Project-3-Data-Warehouse-AWS

  Project 3 - Data Engineering Nanodegree

  awsaws-redshiftudacity-nanodegreedata-engineeringaws-s3
  Language:Python 4

• eduardofb / redshift-create-manifest

  Redshift script to create a MANIFEST file recursively

  redshiftredshift-manifestaws-redshift
  Language:Python 3

• FedericoSerini / DEND-Project-5-Data-Pipelines

  Project 5 - Data Engineering Nanodegree

  awsaws-s3aws-redshiftudacity-nanodegreedata-engineeringdata-pipelinesapache-airflow
  Language:Python 3

• Huyen-P / DE_DWH_AWS_S3_RedShift

  building etl pipelines to migrate music json data/ metadata files (semi-structured data) into a relational database stored in AWS Redshift cluster

  datawarehousevpcpythoncloudshellcmdlineaws-redshiftsql
  Language:Python 3

• lregnier / slick-amazon-redshift

  A quick example of how to load data from Amazon S3 into Amazon Redshift using Redshift's COPY command through Slick

  scalaslickamazonawsaws-redshiftredsfhitcopyaws-s3s3
  Language:Scala 3

• mikecerton / The-Retail-ELT-Pipeline-End-To-End

  This project designs and implements an ETL pipeline using Apache Airflow (Docker Compose) to ingest, process, and store retail data. AWS S3 acts as the data lake, AWS Redshift as the data warehouse, and Looker Studio for visualization. [Data Engineer]

  aws-redshiftaws-s3data-engineeretl-pipelinelooker-studioapache-airflow
  Language:Python 3

• BhawnaMehbubani / Kafka-Spark-Redshift-Streaming-Data-Ingestion-Project

  This project is a real-time data pipeline designed for ingesting, processing, and storing telecom call records. It integrates Apache Kafka, Apache Spark Streaming, and AWS Redshift to handle large volumes of streaming data in near real-time. The pipeline is containerized with Docker Compose, enabling easy deployment, scalability, and modularity.

  apache-kafkaapache-sparkaws-redshiftdockerspark-streaming
  Language:Python 2

• DimaKuriptya / RedditETL

  This project provides a comprehensive data pipeline solution to extract, transform, and load (ETL) Reddit data into a Redshift data warehouse. The pipeline leverages a combination of tools and services including Apache Airflow, Celery, PostgreSQL, Amazon S3, AWS Glue, Amazon Athena, and Amazon Redshift.

  airflowaws-athenaaws-glueaws-redshiftaws-s3celerydockerpandaspostgresqlpythonredis
  Language:Python 2

• DivineSamOfficial / Banking-Data-Warehouse-Pipeline

  Banking Data Warehouse Pipeline

  awsaws-glueaws-redshiftdata-engineering-pipelinedatawarehousingdbtpython
  Language:Python 2

• Hanan-Nawaz / FlightTragedyAnalysis

  Flight Tragedy Analysis is a comprehensive data analysis project focused on examining aviation accidents and incidents from 1905 to 2009. This project provides users with valuable insights into historical plane crashes and their associated data.

  airplane-crashesawsaws-redshiftaws-s3data-engineeringetlkagglepower-bipythonsqlpsycopg2postgresql
  Language:Jupyter Notebook 2