Foundation Model Inference

Foundation Model Inference's repositories

Running large language models on a single GPU for throughput-oriented scenarios.

Language:PythonApache-2.09037 105 80

[NeurIPS'23] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.

Language:Python296 5 27

Language:Python232 6 27