kiminh / imitater

Imitate OpenAI with Local Models

Imitater

A unified language model server built upon vllm and infinity.

Usage

Install

pip install -U imitater

Launch Server

python -m imitater.service.app -c config/example.yaml

Note

Chat template is required for the chat models.

Use export USE_MODELSCOPE_HUB=1 to download model from modelscope.

Test Server

python tests/test_openai.py -c config/example.yaml

Roadmap

• Response choices.
• Rerank model support.