Local LLM as backend for DemoGPT agent
paluigi opened this issue · comments
Is your feature request related to a problem? Please describe.
Using local LLMs instead than OpenAI API as backend
Describe the solution you'd like
Create a DemoGPT agent from a locally available model (ideally, a quantized Llama2 model via llama-cpp-python
Describe alternatives you've considered
If that' s already possible, a guide or some instruction about how to do it would be greatly appreciated!
Additional context
NA
Hi @paluigi,
Thank you for highlighting this feature request. We truly value your feedback and are always eager to improve DemoGPT based on our community's suggestions.
At present, our primary focus is enhancing DemoGPT's capabilities by adding more tools. That said, integrating local models is definitely on our roadmap, and Llama2 is indeed at the top of our list for such integrations.
We appreciate your input and dedication to the growth of DemoGPT. 🙏 Stay tuned for updates!
Hi @paluigi
You might consider referring to the implementation here, which utilizes Fastchat to encapsulate other open-source models for invocation through the OpenAI API. I've employed the chatglm-6b model here, though I'm uncertain whether it supports the llama-cpp model.
Link: https://github.com/chatchat-space/Langchain-Chatchat/blob/master/server/llm_api.py
If necessary, I can submit the code to GitHub. @melih-unsal
Thanks @wusiyaodiudiu , I will have a look to your repo!
Hi @paluigi
I have already forked this project and committed the relevant implementation code for the local LLM,This time, I only committed the part related to the local LLM, so there may be some minor errors. However, I believe the implementation approach is relatively easy to understand and extend.
Newly added files:
demogpt/model_config.py
demogpt/server/llm_api.py
Modified files:"
demogpt/app.py
demogpt/model.py