Narcasserun

spark

Apache Spark - A unified analytics engine for large-scale data processing

skywalking

APM, Application Performance Monitoring System

horovod

Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.

infracost

Cloud cost estimates for Terraform in pull requests💰📉 Shift FinOps Left!

soar

SQL Optimizer And Rewriter

keda

KEDA is a Kubernetes-based Event Driven Autoscaling component. It provides event driven scale for any container running in Kubernetes

skypilot

SkyPilot: Run LLMs, AI, and Batch jobs on any cloud. Get maximum savings, highest GPU availability, and managed execution—all with a simple interface.

arrow-datafusion

Apache DataFusion SQL Query Engine

deequ

Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.

AutoSpotting

Saves up to 90% of AWS EC2 costs by automating the use of spot instances on existing AutoScaling groups. Installs in minutes using CloudFormation or Terraform. Convenient to deploy at scale using StackSets. Uses tagging to avoid launch configuration changes. Automated spot termination handling. Reliable fallback to on-demand instances.

TransmogrifAI

TransmogrifAI (pronounced trăns-mŏgˈrə-fī) is an AutoML library for building modular, reusable, strongly typed machine learning workflows on Apache Spark with minimal hand-tuning

crane

Crane is a FinOps Platform for Cloud Resource Analytics and Economics in Kubernetes clusters. The goal is not only to help users to manage cloud cost easier but also ensure the quality of applications.

gctoolkit

Tool for parsing GC logs

gluten

Gluten: Plugin to Double SparkSQL's Performance

SparkCTR

CTR prediction model based on spark(LR, GBDT, DNN)

blaze

Blazing-fast query execution engine speaks Apache Spark language and has Arrow-DataFusion at its core.

metorikku

A simplified, lightweight ETL Framework based on Apache Spark

katalyst-core

Katalyst aims to provide a universal solution to help improve resource utilization and optimize the overall costs in the cloud. This is the core components in Katalyst system, including multiple agents and centralized components

sparklint

A tool for monitoring and tuning Spark jobs for efficiency.

flink-sql-lineage

The Lineage Analysis system for FlinkSQL supports advanced syntax such as Watermark, UDTF, CEP, Windowing TVFs, and CTAS.

delight

A Spark UI and Spark History Server alternative with CPU and Memory metrics! Delight is free, cross-platform, and open-source.

compass

Compass is a task diagnosis platform for bigdata

gazelle_plugin

Native SQL Engine plugin for Spark SQL with vectorized SIMD optimizations.

spark-metrics

Spark metrics related custom classes and sinks (e.g. Prometheus)

eks-spark-benchmark

Performance optimization for Spark running on Kubernetes

bestconf

A tool automatically improving the performance of large-scale systems by finding better configuration settings

spark-memory

A tool to get better debug info on spark's memory usage

sql-calculator

这是一个基于 TiDB MySQL 语法解析器的一个工具集，支持1. SQL 指纹（sql fingerprint）；2. 数据库库表对比（sql diff）： 对比两个数据库的库表差异，并生成源库到目标库对应的差异（ DDL） 语句。

gceasy

SparkConfigurationsAutotuning