This repo contains a sample for deploying the Llama-2 conversational AI model on RunPod, to quickly spin up an inference server.
The main steps are:
- Install the RunPod Python SDK
- Authenticate with your RunPod API key
- Launch a GPU pod with the Llama container
- Make requests to the pod's endpoint to generate text
The notebook provides examples using:
- The RunPod API directly
- The
requestslibrary - The Hugging Face
text-generationclient
It shows both synchronous requests and streaming.
To use the notebook:
- Clone this repo
- Install the requirements
- Add your RunPod API key
- Run the notebook cells to launch a pod and make requests
The pod will remain running until terminated, so you can experiment with different prompts.
When finished, run the last cell to terminate the pod and avoid continued charges.
The notebook can be adapted by:
- Using different GPU types or regions
- Launching multiple GPUs for faster inference
- Modifying the prompt formatting
- Switching Llama models like
TheBloke/Llama-7b-chat
See the RunPod and Hugging Face docs for more options.
This repo contains a sample for deploying the large 13 billion parameter Llama-2-13b model on Replicate using LangChain to build a conversational agent.
The main steps are:
- Install LangChain and Replicate SDKs
- Create Replicate account and set API token
- Import Llama-2-13b model
- Initialize a LangChain agent with the Replicate LLM
- Run conversations by calling the agent
To run the conversational agent:
- Clone this repo
- Install requirements
- Set your Replicate API token
- Run the cells to deploy the model and initialize the agent
- Call
agent_chain.run()to have a conversation
Stop the model when finished to avoid charges.
The agent can be customized by:
- Using different Llama models
- Modifying the temperature, top-p, etc
- Adding different tools like a Human or SQL tool
- Switching agent architectures
See the LangChain docs for options.