7thMarch / Nicol

Nicol is an open-source web service, developed using the Kotlin programming language, that enables streaming Server Stream Events and supports CPU inference for the port of llama2 model.

Nicol

Nicol is an open-source web service, developed using the Kotlin programming language,

that enables streaming Server Stream Events and supports CPU inference for the port of llama2.c model.

🌟Features

• 🤖 Local Model: A Kotlin port of llama2.c by @karpathy .
• 🎢 Server Stream Event: Implemented a streaming inference interface similar to OpenAI's style .
• 🔒 Authorization: Provides basic interface token validation functionality based on the Spring Framework.

📸 Snapshot

Showcasing in Web Demo	Server Stream Event Restful Api

🚀 Quick Start

Note
**大陆 Ip 请保证网络能够正常访问 Github && Hugging face.
China Ip should make sure your network can access Github && Hugging face properly.

1 Deploy

git clone https://github.com/WackyGem/Nicol.git

cd Nicol

docker-compose up

2 Web

// web demo
http://127.0.0.1:8005/

// openapi
http://127.0.0.1:9005/swagger-ui/index.html

🤝 Thanks

• 🤖 llama2.c : Inference a baby Llama 2 model in pure C .
• 💡 jimmer : A revolutionary ORM framework for both java and kotlin, and a complete integrated solution.

---

Copyright © 2023 Wacky Gem .
This project is MIT licensed.