emr-cluster

There are 5 repositories under emr-cluster topic.

• goodreads_etl_pipeline

  san089 / goodreads_etl_pipeline

  An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

  etl-pipelineetl-frameworksparkapache-sparkapache-airflowairflowredshiftemr-clusterlivys3warehousedata-lakeschedulerdata-migrationdata-engineeringdata-engineering-pipelinepythongoodreads-data-pipelineairflow-dagetl-job
  Language:Python 1261

• RubensZimbres / Repo-2019

  BERT, AWS RDS, AWS Forecast, EMR Spark Cluster, Hive, Serverless, Google Assistant + Raspberry Pi, Infrared, Google Cloud Platform Natural Language, Anomaly detection, Tensorflow, Mathematics

  aws-rdsanomaly-detectiongoogleassistantgooglespeechkeras-tensorflowsql-serverraspberry-pi-3tensorflowmathematicamathewolfram-mathematicaaws-emr-clusterspysparkhiveqlemr-clusterbert-modelbert
  Language:Jupyter Notebook 136

• aws-samples / aws-dbs-refarch-datalake

  Reference Architectures for Datalakes on AWS

  data-lakedata-analyticsamazon-emringest-dataemr-clustergluehive-metastoredata-catalogdata-transformation
  Language:HTML 75

• immu0001 / Udacity-Data-Engineer-nanodegree

  Classwork projects and home works done through Udacity data engineering nano degree

  sparkdata-analysisbig-dataetldata-pipelinesclassworkairflow-dagsdata-sciencedata-lake-analyticss3-bucketemr-clusterredshiftdata-engineering-pipeline
  Language:Jupyter Notebook 72

• cloudposse / terraform-aws-emr-cluster

  Terraform module to provision an Elastic MapReduce (EMR) cluster on AWS

  hcl2emremr-clusteremrfsemr-notebooksterraformterraform-modulesterraform-moduleterraform-awshadoophiveprestospark
  Language:HCL 71

• dacort / demo-code

  Bits of code I use during live demos

  emr-clusteraws-emremr-notebooksaws-cloudformationaws-cloudformation-templatesaws-athenaamazon-athenaamazon-emrlive-demos
  Language:Jupyter Notebook 28
• pyspark-on-aws-emr

  Wittline / pyspark-on-aws-emr

  The goal of this project is to offer an AWS EMR template using Spot Fleet and On-Demand Instances that you can use quickly. Just focus on writing pyspark code.

  awsemr-clusteraws-emrpythonsparkpysparkbig-data-analyticsbig-datadataengineeringwordcloud-generatorec2-spotec2-spot-instances
  Language:Python 25

• camposvinicius / aws-etl

  This is an ETL application on AWS with general open sales and customer data that you can find here: https://github.com/camposvinicius/data/blob/main/AdventureWorks.zip, it's a zipped file with some .csvs inside that we will apply transformations.

  sparkkubernetesairflowawsargocdathenaemrpysparkgluecatalogdatabasepostgresrdsetlpipelinedatadata-engineeremr-cluster
  Language:Smarty 15

• dhiraa / spark-tpcds

  Apache Spark TPC-DS benchmark setup with EMR launch setup

  apachespark-sqlbenchmarkingemr-clusterawspulumi-aws
  Language:Smarty 14

• maelfabien / Cassandra-GDELT-Queries

  A Cassandra Architecture for GDELT Database 🌍

  big-datagdeltgdelt-eventsgdelt-datascalasparkawszeppelinarchitecturecassandraemr-cluster
  Language:Shell 11

• Signiant / dynamodb-emr-exporter

  Uses EMR clusters to export dynamoDB tables to S3 and generates import steps

  emr-clusterdockerawsdynamodbdynamodb-backups
  Language:Shell 11

• xianwill / spark-boilerplate

  A boilerplate for spark projects with docker support for local development and scripts for emr support.

  sparkboilerplateapache-sparkdockeremremr-cluster
  Language:Scala 9
• Batch-ETL-with-AWS-EMR-and-MWAA

  anthonywong611 / Batch-ETL-with-AWS-EMR-and-MWAA

  Create a data pipeline on AWS to execute batch processing in a Spark cluster provisioned by Amazon EMR. ETL using managed airflow: extracts data from S3, transform data using spark, load transformed data back to S3.

  aws-cloudformationemr-clusters3-bucketairflowbatch-processing
  Language:Python 8

• minhky2185 / healthcare_data_pipeline

  An end-to-end data pipeline for building Data Lake and supporting report using Apache Spark.

  analyticsbig-datadatadata-engineeringdata-engineering-pipelinedata-lakeemr-clustermysqlpostgresqlpowerbirds-mysqlrds-postgress3sparkapache-sparkspark-clustervisualizationhealthcare-data
  Language:Python 7

• sjmiller8182 / Warehousing-Stock-Tweet-Data

  A large-scale data framework that will enable us to store and analyze financial market data and drive future predictions for investment.

  big-datahiveawsstock-pricestweetshadoopemr-clusteraws-emrnysenasdaqtwitterdata-warehousestar-schemaspython3snowflake-schemawarehousing-stock-data
  Language:TSQL 7

• airscholar / EMR-for-data-engineers

  This project demonstrates the use of Amazon Elastic Map Reduce (EMR) for processing large datasets using Apache Spark. It includes a Spark script for ETL (Extract, Transform, Load) operations, AWS command line instructions for setting up and managing the EMR cluster, and a dataset for testing and demonstration purposes.

  apache-sparkawsaws-s3emr-cluster
  Language:Python 4

• bdoepf / aws-emr-prometheus

  awsprometheusemremr-clusterapache-sparkapache-flinkapache-hadoop
  Language:HCL 3

• fermat01 / ETL-Data-Pipeline-using-AWS-EMR-Spark-Glue-Athena

  ETL Data pipeline using aws services

  awsaws-athenaaws-ec2aws-glue-crawleremr-cluster
  Language:Python 3

• HarshadRanganathan / aws-emr-launcher

  Generic python library that enables to provision emr clusters with yaml config files (Configuration as Code)

  awsaws-emremr-cluster
  Language:Python 3

• yennanliu / spark_emr_dev

  Collection of code for submitting Spark/Hadoop/Hive/Pig tasks to EMR (AWS Elastic MapReduce) | #DE

  emremr-clusterhadoophadoop-mapreducehivepigsparkspark-etlathenas3scalapysparkmapreducecluster
  Language:Scala 3

• AchilleasKn / AWS_EMR_cluster

  Amazon EMR for Data Science

  awsemremr-clusterdatasciencebigdataapache-sparkhadoopawsclijupytersparkpyspark
  2

• alex-ber / docker-hive

  EMR 5.25.0 cluster single node Hadoop docker image. With Amazon Linux, Hadoop 2.8.5 and Hive 2.3.5

  hadoop-dockerhivedockerdocker-composedockerfiledocker-imagehadoop-hdfshadoop-mapreducehadoop-clusterhadoop-ecosystemhadoop-frameworkhadoop-filesystemyarn-hadoop-clusteryarnhiveserver2dockerfilesdocker-imageshadoopemremr-cluster
  Language:Shell 2

• JohnnyLVP / Project-Standar-Documentation

  This repository contains a definition of standar structure for Machine Learning and Data Pipelines Projects

  standardmachine-learningprojectdocumentationpythonpysparkredshiftboto3awsec2emremr-cluster
  Language:Python 2

• mikeacosta / florasense

  Orchestrating Cloud ETL Workloads

  awskinesis-streamredshiftapache-sparklambda-functionsemr-clusteretl-pipelinecloudformationstep-functionsdata-lakedata-warehouseredshift-spectrum
  Language:Python 2

• sepulworld / serverless-aws-emr-boilerplate

  Event driven EMR via Serverless

  emr-clusteremrserverless-frameworkserverlessaws-lambdapython3aws-apigatewayaws-sns
  Language:Python 2

• UdeshikaDissa / BigData-MapReduce

  This BigData study intends to identify the most revenue-generating Taxi zones in New York City for the year 2019. Three MapReduce algorithms were developed and their performance was analyzed on different size of input datasets and different size clusters in EMR.

  hadoop-mapreduceemr-clusterbig-datajavain-mapper-combineraws
  Language:Java 2

• amrelauoty / Sparkify-Datalake-AWS

  Data Engineering Expert Nanodegree - Data Lake on AWS using Spark and S3

  awsaws-cliaws-s3emremr-clusters3s3-buckets3-storagesparkspark-sql
  Language:Jupyter Notebook 1

• arfatmateen / Data_Lake_and_ETL_Pipeline_on_AWS_using_Spark

  Database Schema & ETL pipeline for Song Play Analysis | Bosch AI Talent Accelerator Scholarship Program

  awsemr-clusteretl-pipelinejupyter-notebookpysparkpythons3-bucketsql
  Language:Python 1

• EddieAmaitum / NYC-Yellow-Taxi-DataOps-with-AWS-Analyzing-TLC-Datasets

  Performed business operations using Big data technologies: AWS EMR, AWS RDS (MySQL), Hadoop, Apache Scoop, Apache HBase, MapReduce

  awsemr-clusterhadoophbaselinux-shellpythonrds-databasesqoopmapreduce-jobs
  Language:Python 1
• AWS-EMR-APACHE-SPARK

  jpb111 / AWS-EMR-APACHE-SPARK

  Guide: Executing a python script on AWS EMR for big data analysis.

  awsaws-ec2aws-s3emr-clusterpandaspysparkpythonemr-serverless
  Language:Python 1

• longNguyen010203 / Spark-Processing-AWS

  👷🌇 Set up and build a big data processing pipeline with Apache Spark, 📦 AWS services (S3, EMR, EC2, IAM, VPC, Redshift) Terraform to setup the infrastructure and Integration Airflow to automate workflows🥊

  apache-sparkawsaws-ec2aws-s3emr-clusteriampysparkredshiftspark-masterspark-workeraws-servicescloud-computingdata-pipelinespark-clusterterraformapache-airflow
  Language:Python 1

• manaswikamila05 / Public-Clickstream-Data-Analysis

  Used a public clickstream dataset of a cosmetics store to extract data and gather insights. Launched an EMR 5.29.0 cluster that utilizes Hive services and used optimized hive queries to improve their sales by identifying customer behavior.

  hivehadoophdfss3emr-clusterhiveqlpartitioning
  1

• Morgan-Sell / usa-tourism-etl

  Coalesced and transformed various data sources to create a comprehensive data lake for the USA tourism sector.

  etl-pipelinedata-engineeringpythonsparkemr-clusterawsdata-lake
  Language:Jupyter Notebook 1

• nileshsingal / PUBG-DATA-ANALYSIS

  Player Unknown's Battlegrounds (PUBG), is a first person shooter game where the goal is to be the last player standing. You are placed on a giant circular map that shrinks as the game goes on, and you must find weapons, armor, and other supplies in order to kill other players / teams and survive.

  bigdataemr-clusters3-bucketsparkhiveapi-gatewayaws-lambdaaws-cloudformationtableau
  Language:Python 1
• Recipe_Recommender_Asssignment_EDA_Using_PySpark

  shantamgarg24 / Recipe_Recommender_Asssignment_EDA_Using_PySpark

  Used Amazon AWS and PySpark to solve this EDA assignment

  amazon-web-servicesapache-sparkdata-scienceemr-clusterexploratory-data-analysispysparkrecommender-system
  Language:Jupyter Notebook 1

• Tanay0510 / Data-Lake-with-Spark

  Load data from S3, process the data into analytics tables using Spark and load them back into S3. Deployed this Spark process on a cluster using AWS EMR

  s3emr-clusteretl-pipelinesparkdatalake
  Language:Python 1