Support Alpaca and Llama models
samheutmaker opened this issue · comments
Autodoc is currently reliant on OpenAI for access to cutting-edge language models. Going forward, we would like to support models running locally or at providers other than OpenAI, like Llama, or Alpaca. This gives developers more control over how their code is indexed, and allows indexing of private code that cannot be shared with OpenAI.
This is a big undertaking that will be an on-going process. A few thoughts for someone who wants to get starting hacking on this.
- It would be nice to be able to configure Autodoc with a LangChain LLM via the Autodoc config file. This would allow for complete control over how an LLM is configured.
- It seems like a lot of people are using llamma.cpp to run llamma locally. It may be worth using this as a starting point to support other models.
This issue is high priority. If you're interesting in working on it, please reach out.
Ideally the model configuration stuff should be abstracted out to something like langchain, it's too bad there no TS port for it yet.
Yes there is, we already use it in Autodoc. https://github.com/hwchase17/langchainjs
Maybe consider use https://github.com/rustformers/llama-rs in that case?
and also https://github.com/sobelio/llm-chain for langchain.
have you seen this project? https://github.com/microsoft/semantic-kernel
It might help with this task....
Semantic Kernel (SK) is a lightweight SDK enabling integration of AI Large Language Models (LLMs) with conventional programming languages. The SK extensible programming model combines natural language semantic functions, traditional code native functions, and embeddings-based memory unlocking new potential and adding value to applications with AI.
SK supports prompt templating, function chaining, vectorized memory, and intelligent planning capabilities out of the box.