georgehu0815

dbt-spark

dbt-spark contains all of the code enabling dbt to work with Apache Spark and Databricks

airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

awesome-chatgpt-prompts

This repo includes ChatGPT prompt curation to use ChatGPT better.

ebm2onnx

A tool to convert EBM models to ONNX

gpt-2-Pytorch

Simple Text-Generator with OpenAI gpt-2 Pytorch Implementation

lm-human-preferences

Code for the paper Fine-Tuning Language Models from Human Preferences

trl

Train transformer language models with reinforcement learning.

transformers_tasks

⭐️ NLP Algorithms with transformers lib. Supporting Text-Classification, Text-Generation, Information-Extraction, Text-Matching, RLHF, SFT etc.

batch-processing-gateway

The gateway component to make Spark on K8s much easier for Spark users.

feathr

Feathr – A scalable, unified data and AI engineering platform for enterprise

spark-jobserver

REST job server for Apache Spark

delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

uvicorn-gunicorn-fastapi-docker

Docker image with Uvicorn managed by Gunicorn for high-performance FastAPI web applications in Python with performance auto-tuning.

conduit

Conduit streams data between data stores. Kafka Connect replacement. No JVM required.

spark

Apache Spark - A unified analytics engine for large-scale data processing

hopsworks

Hopsworks - Data-Intensive AI platform with a Feature Store

delight

A Spark UI and Spark History Server alternative with CPU and Memory metrics! Delight is free, cross-platform, and open-source.

dask-jobqueue

Deploy Dask on job schedulers like PBS, SLURM, and SGE

dask-kubernetes

Native Kubernetes integration for Dask

yunikorn-scheduler-interface

Apache YuniKorn Scheduler Interface

yunikorn-core

Apache YuniKorn Core

yunikorn-web

Apache YuniKorn Web UI

dashboard

General-purpose web UI for Kubernetes clusters

winutils

winutils.exe hadoop.dll and hdfs.dll binaries for hadoop windows

docker-spark-iceberg

iceberg-rest-image

Simple project to expose a catalog over REST using a Java catalog backend

manageability-toolkits

Sample code to get quickly on-boarded to common Azure manageability tools and platforms like Azure Monitor.

recommenders

Best Practices on Recommendation Systems

azure-event-hubs-spark

Enabling Continuous Data Processing with Apache Spark and Azure Event Hubs

azure-machine-learning-bicep

A set of Bicep templates for Azure Machine Learning.