Chatbots have always been WoW stuff!! The recent evidence is: ChatGPT.
Now that they are more human-like with the latest LLMs (Large Language Models). But these LLMs are Pretrained on their own (HUGE) data. Mere mortals dont have any ways ($$, time, expertise) to train own LLMs. Some do have facility to get fine-tuned on custom corpus, but limited. This repo explores this further. Wish to build that end-to-end MLOps for fine-tuning LLMs.
Goal: build fine-tuning on LLMs on own corpus:
- corpus can be documents: FAQs, manuals medical papers, etc (many tutorials are available to do this via Vector Databases)
- corpus can be tables, so need SQL/BI conversion from natural language
- corpus can be graphs: social networks, need conversion to GraphGPT, cypher
- Open source: LangChain using HuggingFace free models (Open AI models are cheap also), for local, data-secure solution
- Google Cloud: End-yo-end VertexAI MLOps, easy deployment, for enterprise internal solution.
- Building the Future with LLMs, LangChain, & Pinecone
- LangChain for Gen AI and LLMs - James Briggs
- Finetuning GPT-3 David Shapiro ~ AI
- Build overpowered AI apps with the OP stack (OpenAI + Pinecone)
- Learn about AI Language Models and Reinforcement Learning Kamalraj M M
- GPT-4 & LangChain Tutorial: How to Chat With A 56-Page PDF Document (w/Pinecone)
- LangChain - Data Independent
- Practical AI by Ramsri NLP, GPT, MicroSaaS
- Dhramesh Shah ChatSpot, ChatUX