aws-emr
There are 9 repositories under aws-emr topic.
adornes / spark_python_ml_examples
Spark 2.0 Python Machine Learning examples
Language:Python 95Spark 2.0 Scala Machine Learning examples
Language:Scala 77- Language:Python 67
- Language:Jupyter Notebook 28
abdullahkhawer / aws-auto-terminate-idle-emr
AWS Auto Terminate Idle AWS EMR Clusters Framework is an AWS based solution using AWS CloudWatch and AWS Lambda using a Python script that is using Boto3 to terminate AWS EMR clusters that have been idle for a specified period of time.
Language:Python 27- pyspark-on-aws-emr
Wittline / pyspark-on-aws-emr
The goal of this project is to offer an AWS EMR template using Spot Fleet and On-Demand Instances that you can use quickly. Just focus on writing pyspark code.
Language:Python 24 terraform-aws-modules / terraform-aws-emr
Terraform module to create AWS EMR resources 🇺🇦
Language:HCL 21- aws-data-pipeline
ismaildawoodjee / aws-data-pipeline
A batch processing data pipeline, using AWS resources (S3, EMR, Redshift, EC2, IAM), provisioned via Terraform, and orchestrated from locally hosted Airflow containers. The end product is a Superset dashboard and a Postgres database, hosted on an EC2 instance at this address (powered down):
Language:Python 17 - Language:Python 16
- Language:Python 15
AWS-Big-Data-Projects / Analysing-Census-Data-using-aws
Use aws-emr and aws-redshift to analyse dataset of adult census of USA
Run a Spark job within Amazon EMR
Language:Java 12ychantit / airflow_aws_utils
A collection of airflow sample workflows for data processing on aws
Language:Python 12xonai-computing / xonai-dashboard
A Grafana-based application to assist Big Data infrastructure optimization initiatives where Spark applications are a dominant cost driver
Language:Python 11- Language:Dockerfile 10
jkoth / Data-Lake-with-Spark-and-AWS-S3
Create Data Lake on AWS S3 to store dimensional tables after processing data using Spark on AWS EMR cluster
Language:Python 9linghaol / CommunityDetection-Spark-AWS
A Spark application, written in Python, to figure out strongly connected components with Bi-directional Label Propagation algorithm. This project implemented an 1.3GB Twitter network dataset on AWS EMR cluster.
Language:Python 9- cookiecutter-pyspark-cloud
daniel-cortez-stevenson / cookiecutter-pyspark-cloud
A cookiecutter template for working with PySpark on AWS EMR
Language:Python 8 Nerdward / batch_gh_archive
Data Engineering Project with Terraform, Spark, AWS, Docker, Airflow and other tools
Language:Python 8sjmiller8182 / Warehousing-Stock-Tweet-Data
A large-scale data framework that will enable us to store and analyze financial market data and drive future predictions for investment.
Language:TSQL 7Spark 2.0 R/SparkR Machine Learning examples
Language:R 6wingkwong / aws-playground
My AWS Playground
Language:Python 6felipeazucares / Airflow-EMR-Redshift
EMR + Hadoop to Redshift ELT workflow using spark steps API and orchestrated by Apache-Airflow, which ingests disparate datasets focused around 7Gb of I94 arrivals information to produce a simple star schema in Redshift
Language:Python 4pratikbarjatya / spark-walmart-data-analysis-exercise
Data Analysis Exercise over Walmart Stock
Language:Jupyter Notebook 4abhibalani / emr_lambda
Lambda to start EMR and run a map reduce job
Language:Python 3HarshadRanganathan / aws-emr-launcher
Generic python library that enables to provision emr clusters with yaml config files (Configuration as Code)
Language:Python 3JainTanisha / MapReduce-Analysis-on-Amazon-Food-Review-Data
MapReduce Analysis on Amazon Food Review Dataset (Big-Data)
khushal2405 / Daily-Incremental-load-ETL-pipeline-for-Ecommerce-company-using-AWS-Lambda-and-Apache-airflow
Daily Incremental load ETL pipeline for Ecommerce company using AWS Lambda and AWS EMR cluster, Deployed using Apache airflow in a docker container.
Language:Python 3shinde-chandrakant / BigData-Ops-on-TLC-Yellow-Taxi
Analysed New York City's Yellow taxi data set with Big Data tools such as Hadoop, HBase, Sqoop, MapReduce and AWS Cloud Infrastructure.
Language:Python 3AuFeld / AWS_MWAA_With_Step_Functions
Build modern workflows with AWS MWAA, AWS Step Functions, AWS Glue, and AWS EMR
Language:Python 2dhruv007patel / Impact-of-Covid-19-on-Aviation-Industry
This project analyzes the correlation between COVID-19 and the US aviation industry. By studying data on passenger/freight traffic and delays alongside COVID-19 trends, it provides insights into airline and passenger responses. The findings help airlines adapt to the pandemic's impact.
Language:Python 2jonathanAmancioSales / BigData_AWS_EMR_MRJob_DIO
Projeto de processamento distribuído de dados utilizando Python, MRJob e AWS EMR
Language:Python 2micopes / AWS_Datalake
AWS 및 AWS를 이용한 Data Lake 구성 이해
ninjeanne / datastorm
Data Science and Engineering project - Programming for Big Data @ Simon Fraser University (SFU)
Language:Jupyter Notebook 2RahilBalar98 / Covid-19-And-Aviation-Analysis
CMPT 732 Project - Dealt with 3 large scale databases by joining them to analysis the economic impact of Covid-19 on the airline industry. Fetched data using API and stored in AWS S3 that is retrieved by an AWS EMR cluster that does data computation. Queried into AWS Athena and visualized the results on Tableau by implementing static and dynamic dashboards.
Language:Python 2
