mengxr

pyspark-ai

English SDK for Apache Spark

llm-numbers

Numbers every LLM developer should know

LLMsPracticalGuide

A curated list of practical guide resources of LLMs (LLMs Tree, Examples, Papers)

run-notebook

sparkext

Spark DL Inferencing using external frameworks

terraform-provider-databricks

Databricks Terraform Provider

ML-Engineering

Reference code base for ML Engineering, Manning Publications

automl

byteps

A high performance and generic framework for distributed DNN training

joblib-spark

Joblib Apache Spark Backend

koalas

Koalas: pandas API on Apache Spark

IncrementalMoments.jl

Julia package to computes statistics on streams of data

StratifiedRandomSampling

Spark Exercise

node2vec

intellij-jsonnet

Intellij Jsonnet Plugin

spark-salesforce

Spark data source for Salesforce

cloud-custodian

Rules engine for cloud security, cost optimization, and governance, DSL in yaml for policies to query, filter, and take actions on resources

databricks-cli

(Legacy) Command Line Interface for Databricks

drunken-data-quality

Spark package for checking data quality

airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Rserve

Fast, flexible and powerful server providing access to R from many languages and systems

quartz

Code for Quartz Scheduler

ipex-llm

Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, Baichuan, Mixtral, Gemma, Phi, etc.) on Intel CPU and GPU (e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, DeepSpeed, vLLM, FastChat, Axolotl, etc.

SparkADMM

Generic Implementation of Consensus ADMM over Spark

sparklyr

R interface for Apache Spark

photon-ml

A scalable machine learning library on Apache Spark

superset

Apache Superset is a Data Visualization and Data Exploration Platform

tensorframes

[DEPRECATED] Tensorflow wrapper for DataFrames on Apache Spark

graphframes

lazy-linalg

A package full of linear algebra operators for Apache Spark MLlib's linalg package