SambaNova AI Starter Kits

Overview

SambaNova AI Starter Kits are a collection of open-source examples and guides to facilitate the deployment of AI-driven use cases in the enterprise.

To run these examples, you need access to a SambaStudio environment with your models deployed to endpoints. Most code examples are written in Python, though the concepts can be applied in any language.

Questions? Just message us on Discord or create an issue in GitHub. We're happy to help live!

AI Starter Kits - Ecosystem

Data Ingestion & Preparation

🔍 Data Extraction
📊 Web-crawled Data Retriever
🧠 Synthetic Q&A Pair Creation

Model Development & Optimization

🚀 YoDA - Your Data, Your Model Recipe
🎯 Q&A Fine-tuning
💾 SQL Model Fine-tuning
🔬 Fine-tuning Embeddings

Intelligent Information Retrieval

📚 Enterprise Knowledge Retriever
🔎 Search Assistant
🖼️ Image Search
🖼️ Multi-Modal Knowledge Retriver

Advanced AI Capabilities

🧭 CoE Routing
💡 Prompt Engineering
🔀 Performance Benchmarking
🔧 Function Calling

Deployment & Integration

🖥️ Endpoint Creation
🔗 Samba Apps Integration
🐳 Docker Images

Available AI Starter Kits

Benchmarking	This kit evaluates the performance of multiple LLM models hosted in SambaStudio. It offers various performance metrics and configuration options. Users can also see these metrics within a chat interface.
Code Copilot	This example guide shows a simple integration with Continue VSCode and JetBrains extension using SambaNova platforms, to use Sambanova's hosted models as your custom coding assistant.
Complex RAG	Sample implementation of a complex RAG workflow using the SambaNova platform to get answers to questions about your documents. Includes a runnable demo.
CoE jump start	This kit demonstrates how to call SambaNova CoE models using the Langchain framework. The script offers different approaches for calling CoE models, including using Sambaverse, using SambaStudio with a named expert, and using SambaStudio with routing.
Data Extraction	Series of notebooks that demonstrate methods for extracting text from documents in different input formats.
EDGAR Q&A	Example workflow that uses the SambaNova platform to answer questions about organizations using their 10-K annual reports. Includes a runnable local demo and a Docker container to simplify remote deployment.
Enterprise Knowledge Retrieval	Sample implementation of the semantic search workflow using the SambaNova platform to get answers to questions about your documents. Includes a runnable demo.
Fine tuning embeddings	Example workflow for fine-tuning embeddings from unstructured data, leveraging Large Language Models (LLMs) and open-source embedding models to enhance NLP task performance.
Fine tuning SQL	Example workflow for fine-tuning an SQL model for Question-Answering purposes, leveraging Large Language Models (LLMs) and open-source embedding models to enhance SQL generation task performance.
Function Calling	Example of tools calling implementation and a generic function calling module that can be used inside your application workflows.
Image Search	This example workflow shows a simple approach to image search by image description or image similarity. All workflows are built using the SambaNova platform.
Multimodal Knowledge Retriever	Sample implementation of the semantic search workflow leveraging the SambaNova platform to get answers using text, tables, and images to questions about your documents. Includes a runnable demo.
Post Call Analysis	Example workflow that shows a systematic approach to post-call analysis including Automatic Speech Recognition (ASR), diarization, large language model analysis, and retrieval augmented generation (RAG) workflows. All workflows are built using the SambaNova platform.
Prompt Engineering	Starting point demo for prompt engineering using SambaNova's API to experiment with different use case templates. Provides useful resources to improve prompt crafting, making it an ideal entry point for those new to this AISK.
Search Assistant	Sample implementation of the semantic search workflow built using the SambaNova platform to get answers to your questions using search engine snippets, and website crawled information as the source. Includes a runnable demo.
Web Crawled Data Retrieval	Sample implementation of a semantic search workflow built using the SambaNova platform to get answers to your questions using website crawled information as the source. Includes a runnable demo.
YoDA: Your Data Your model	Sample training recipe to train a Language Model (LLM) using a customer's private data.

Get started with SambaNova AI starter kit

Setting your model

Use Sambaverse models (Option 1)

Sambaverse allows you to interact with multiple open-source models. You can view the list of available models and interact with them in the playground.

Please note that Sambaverse's free offering is performance-limited. Companies that are ready to evaluate the production tokens-per-second performance, volume throughput, and 10x lower total cost of ownership (TCO) of SambaNova should contact us for a non-limited evaluation instance.

Begin by creating a Sambaverse account, then get your API key from the username button. Use the available models.

Deploy your model in SambaStudio (Option 2)

Begin by deploying your LLM of choice (e.g. Llama 2 13B chat, etc) to an endpoint for inference in SambaStudio. Use either the GUI or CLI, as described in the SambaStudio endpoint documentation.

Integrate your model in the starter kit

Integrate your LLM deployed on SambaStudio with this AI starter kit in two simple steps:

1. Clone this repo

  git clone https://github.com/sambanova/ai-starter-kit.git

2. Update API information for the SambaNova LLM

These are represented as configurable variables in the environment variables file in sn-ai-starter-kit/.env.

SambaStudio deployed model

For example, enter an endpoint with the URL "https://api-stage.sambanova.net/api/predict/nlp/12345678-9abc-def0-1234-56789abcdef0/456789ab-cdef-0123-4567-89abcdef0123" in the env file (with no spaces) as:

BASE_URL="https://api-stage.sambanova.net"
PROJECT_ID="12345678-9abc-def0-1234-56789abcdef0"
ENDPOINT_ID="456789ab-cdef-0123-4567-89abcdef0123"
API_KEY="89abcdef-0123-4567-89ab-cdef01234567"

Sambaverse model

Enter a Sambaverse API key, for example "456789ab-cdef-0123-4567-89abcdef0123", in the env file (with no spaces) as:

SAMBAVERSE_API_KEY="456789ab-cdef-0123-4567-89abcdef0123"

3. Update API information for SambaNova Embeddings model (optional).

You can use SambaStudio E5 embedding model endpoint instead of using default in cpu HugginFace embeddings to increase inference speed, follow this guide to deploy your SambaStudio embedding model

be sure to set batch size model parameter to 32

Update API information for the SambaNova embedding endpoint. These are represented as configurable variables in the environment variables file in the root repo directory sn-ai-starter-kit/.env. For example, an endpoint with the URL "https://api-stage.sambanova.net/api/predict/nlp/12345678-9abc-def0-1234-56789abcdef0/456789ab-cdef-0123-4567-89abcdef0123" would be entered in the env file (with no spaces) as:

EMBED_BASE_URL="https://api-stage.sambanova.net"
EMBED_PROJECT_ID="12345678-9abc-def0-1234-56789abcdef0"
EMBED_ENDPOINT_ID="456789ab-cdef-0123-4567-89abcdef0123"
EMBED_API_KEY="89abcdef-0123-4567-89ab-cdef01234567"

Note that using different embedding models (cpu or sambastudio) may change the results, and change the way they are set and their parameters

with CPU Huggingface embeddings:

           embeddings = HuggingFaceInstructEmbeddings(
               model_name="hkunlp/instructor-large",
               embed_instruction="",
               query_instruction="Represent this sentence for searching relevant passages:",
               encode_kwargs={"normalize_embeddings": True},
           )

with Sambastudio embeddings:

embeddings = SambaNovaEmbeddingModel()

4. Run the desired starter kit

Go to the README.md of the starter kit you want to use and follow the instructions. See Available AI Starter Kits.

Use Sambanova's LLMs and Langchain wrappers

LLM Wrappers

Set your environment as shown in integrate your model.

Using Sambaverse LLMs

Import the samabanova_endpoint langchain wrapper in your project and define your SambaverseEndpoint LLM:

from utils.sambanova_endpoint import SambaverseEndpoint

load_dotenv('.env')

llm = SambaverseEndpoint(
    sambaverse_model_name="Meta/llama-2-7b-chat-hf",
    model_kwargs={
      "do_sample": False,
      "temperature": 0.0,
      "max_tokens_to_generate": 512,
      "select_expert": "llama-2-7b-chat-hf"
      },
)

Use the model

llm.invoke("your prompt")

Using Sambastudio LLMs

Import the samabanova_endpoint langchain wrapper in your project and define your SambaNovaEndpoint LLM:

from utils.sambanova_endpoint import SambaNovaEndpoint

load_dotenv('.env')

llm = SambaNovaEndpoint(
    model_kwargs={
      "do_sample": False,
      "max_tokens_to_generate": 512,
      "temperature": 0.0
      },
)

Use the model

llm.invoke("your prompt")

See utils/usage.ipynb for an example.

Embedding Wrapper

Import the samabanova_endpoint langchain wrapper in your project and define your SambaNovaEmbeddingModel embedding:

from utils.sambanova_endpoint import SambaNovaEmbeddingModel

load_dotenv('.env')

embedding = SambaNovaEmbeddingModel()

Use your embedding model in your langchain pipeline

See utils/usage.ipynb for an example.

Setting Up Your Environment

There are two approaches to setting up your environment for the AI Starter Kits:

Individual Kit Setup (Traditional Method)
Base Environment Setup (Recommended)

1. Individual Kit Setup

Each starter kit has its own README.md and requirements.txt file. You can set up a separate virtual environment for each kit by following the instructions in their respective directories. This method is suitable if you're only interested in running a single kit or prefer isolated environments for each project.

To use this method:

Navigate to the specific kit's directory
Create a virtual environment
Install the requirements
Follow the kit-specific instructions

2. Base Environment Setup

For users who plan to work with multiple kits or prefer a unified development environment, we recommend setting up a base environment. This approach uses a Makefile to automate the setup of a consistent Python environment that works across all kits.

Benefits of the base environment approach:

Consistent Python version across all kits
Centralized dependency management
Simplified setup process
Easier switching between different kits

Prerequisites

Poetry: The Makefile will attempt to install Poetry if it's not already installed.
pyenv: The Makefile will attempt to install pyenv if it's not already installed.

What the Base Setup Does

Installs pyenv and Poetry if they are not already installed.
Sets up a Python virtual environment using a specified Python version (default is 3.11.3).
Installs all necessary dependencies for the base environment.
Sets up the parsing service required by some kits.

Setting Up the Base Environment

Install and Set Up the Base Environment:

make all

This command will set up the base ai-starter-kit environment, including installing all necessary tools and dependencies.

Activate the Base Environment:

source .venv/bin/activate

Navigate to Your Chosen Starter Kit:

cd path/to/starter_kit

Within the starter kit there will be instructions on how to start the kit. You can skip the virtual environment creation part in the kits README.md as we've done it here.

Enterprise Knowledge Retriever One Click Deploy: For the EKR Kit specifically we have enabled a one click deploy via makefile

Enterprise Knowledge Retriever (EKR) Setup

To set up and run the Enterprise Knowledge Retriever:

Set Up and Start EKR:

make ekr

This command sets up the EKR environment, starts the parsing service, and launches the EKR application.

Parsing Service Management

For certain kits, we utilise a standard parsing service. To work with this service, following the steps in this section.

Start Parsing Service:

make start-parsing-service

Stop Parsing Service:

make stop-parsing-service

Check Parsing Service Status:

make parsing-status

View Parsing Service Logs:

make make parsing-log

Cleanup

To clean up all virtual environments created by the makefile and stop parsing services run the following command:

make clean

This command removes all virtual environments created with the makefile, stops the parsing service, and cleans up any temporary files.

Troubleshooting

If you encounter issues while setting up or running the AI Starter Kit, here are some common problems and their solutions:

Lock file mismatch

If you see an error message like this:

ERROR
Installing dependencies... Installing dependencies from lock file
pyproject.toml changed significantly since poetry.lock was last generated. Run `poetry lock [--no-update]` to fix the lock file. make: *** [install] Error 1

This means there's a mismatch between your pyproject.toml and poetry.lock files. To resolve this:

Run the following command to update the lock file:
```
poetry lock
```
Then, try running make all again.

Python version issues

If you're having problems with Python versions:

Ensure you have pyenv installed: make ensure-pyenv
Install the required Python versions: make install-python-versions
If issues persist, check your system's Python installation and PATH settings.

Dependency conflicts

If you're experiencing dependency conflicts:

Try cleaning your environment: make clean
Update the lock file: poetry lock --no-update
Reinstall dependencies: make install

Parsing service issues

If the parsing service isn't starting or is behaving unexpectedly:

Check its status: make parsing-status
View its logs: make parsing-log
Try stopping and restarting it: make stop-parsing-service followed by make start-parsing-service

General troubleshooting steps

Ensure all prerequisites (Python, pyenv, Poetry) are correctly installed.
Try cleaning and rebuilding the environment: make clean all
Check for any error messages in the console output and address them specifically.
Ensure your .env file is correctly set up in the ai-starter-kit root with all necessary environment variables.

If you continue to experience issues, please open an issue with details about your environment, the full error message, and steps to reproduce the problem.

Important Notes for Users

Ensure you have sufficient permissions to install software on your system.
The setup process may take several minutes, especially when installing Python versions or large dependencies.
If you encounter any issues during setup, check the error messages and ensure your system meets all prerequisites.
Always activate the base environment before navigating to and running a specific starter kit.
Some kits may require additional setup steps. Always refer to the specific README of the kit you're using.

Note: These AI Starter Kit code samples are provided "as-is," and are not production-ready or supported code. Bugfix/support will be on a best-effort basis only. Code may use third-party open-source software. You are responsible for performing due diligence per your organization policies for use in your applications.