llm-server

There are 0 repository under llm-server topic.

fcn94 / llm_stream_endpoint
Simple LLM Rest API using Rust, Warp and Candle. Dedicated for quantized version of either phi-2 ( default) , Mistral, or Llama. Work using CPU or CUDA
candle llm rust mistral-7b phi-2 api llm-server endpoint streaming quantized llama2 rest rest-api cpu cuda cuda-kernels
Language:Rust 3

fcn94 / llm_stream_endpoint