fuenos01

spark-sql-perf

MapReduce-Performance_Testing

MapReduce performance testing using teragen and terasort

terraform-aws-eks

Terraform module to create Amazon Elastic Kubernetes (EKS) resources 🇺🇦

terragrunt

Terragrunt is a flexible orchestration tool that allows Infrastructure as Code written in OpenTofu/Terraform to scale.

terraform

Terraform enables you to safely and predictably create, change, and improve infrastructure. It is a source-available tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.

pyenv-virtualenv

a pyenv plugin to manage virtualenv (a.k.a. python-virtualenv)

pyenv

Simple Python version management

docker-airflow

Docker Apache Airflow

Miscellaneous

Includes notes on using Apache Spark in general, notes on using Spark for Physics, how to run TPCDS on PySpark, how to create histograms with Spark, tools for performance testing CPUs, Jupyter notebooks examples for Spark, examples for Oracle and other DB systems.

brew

🍺 The missing package manager for macOS (or Linux)

gradle

Adaptable, fast automation for all

phil_stopwatch

flintrock

A command-line tool for launching Apache Spark clusters.

spark-ec2

Scripts used to setup a Spark cluster on EC2

hudi

Upserts, Deletes And Incremental Processing on Big Data.

azkaban

Azkaban workflow manager.

spark-tpcds-datagen

All the things about TPC-DS in Apache Spark

spark-tpc-ds-performance-test

Use the TPC-DS benchmark to test Spark SQL performance

spark-daria

Essential Spark extensions and helper methods ✨😲

spark-style-guide

Spark style guide

awesome-spark

A curated list of awesome Apache Spark packages and resources.

airflow-maintenance-dags

A series of DAGs/Workflows to help maintain the operation of Airflow

airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

powermock

PowerMock is a Java framework that allows you to unit test code normally regarded as untestable.

jvm-profiler

JVM Profiler Sending Metrics to Kafka, Console Output or Custom Reporter

s4cmd

Super S3 command line tool

s3cmd

Official s3cmd repo -- Command line tool for managing S3 compatible storage services (including Amazon S3 and CloudFront).

delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

Spark-The-Definitive-Guide

Spark: The Definitive Guide's Code Repository

scopt

command line options parsing for Scala