ASHNOORSINGH / langsmith-cookbook

LangSmith Cookbook

Welcome to the LangSmith Cookbook — your practical guide to mastering LangSmith. While our standard documentation covers the basics, this repository delves into common patterns and some real-world use-cases, empowering you to optimize your LLM applications further.

This repository is your practical guide to maximizing LangSmith. As a tool, LangSmith empowers you to debug, evaluate, test, and improve your LLM applications continuously. These recipes present real-world scenarios for you to adapt and implement.

Your Input Matters

Help us make the cookbook better! If there's a use-case we missed, or if you have insights to share, please raise a GitHub issue (feel free to tag Will) or contact the LangChain development team. Your expertise shapes this community.

Tracing your code

Tracing allows for seamless debugging and improvement of your LLM applications. Here's how:

• Tracing without LangChain: learn to trace applications independent of LangChain using the Python SDK's @traceable decorator.
• REST API: get acquainted with the REST API's features for logging LLM and chat model runs, and understand nested runs. The run logging spec can be found in the LangSmith SDK repository.
• Customizing Run Names: improve UI clarity by assigning bespoke names to LangSmith chain runs—includes examples for chains, lambda functions, and agents.
• Tracing Nested Calls within Tools: include all nested tool subcalls in a single trace by using run_manager.get_child() and passing to the child callbacks
• Display Trace Links: add trace links to your app to speed up development. This is useful when prototyping your application in its unique UI, since it lets you quickly see its execution flow, add feedback to a run, or add the run to a dataset.

LangChain Hub

Efficiently manage your LLM components with the LangChain Hub. For dedicated documentation, please see the hub docs.

• RetrievalQA Chain: use prompts from the hub in an example RAG pipeline.
• Prompt Versioning: ensure deployment stability by selecting specific prompt versions over the 'latest'.
• Runnable PromptTemplate: streamline the process of saving prompts to the hub from the playground and integrating them into runnable chains.

Testing & Evaluation

Test and benchmark your LLM systems using methods in these evaluation recipes:

Python Examples

• Prompt Iteration Walkthrough: run regression tests to compare multiple prompts on 3 datasets

Retrieval Augmented Generation (RAG)

• Q&A System Correctness: evaluate your retrieval-augmented Q&A pipeline end-to-end on a dataset. Iterate, improve, and keep testing.
• Evaluating Q&A Systems with Dynamic Data: use evaluators that dereference a labels to handle data that changes over time.
• RAG Evaluation using Fixed Sources: evaluate the response component of a RAG (retrieval-augmented generation) pipeline by providing retrieved documents in the dataset

Chat Bots

• Chat Bot Evals using Simulated Users: evaluate your chat bot using a simulated user. The user is given a task, and you score your assistant based on how well it helps without being breaking its instructions.
• Single-turn evals: Evaluate chatbots within multi-turn conversations by treating each data point as an individual dialogue turn. This guide shows how to set up a multi-turn conversation dataset and evaluate a simple chat bot on it.

Extraction

• Evaluating an Extraction Chain: measure the similarity between the extracted structured content and structured labels using LangChain's json evaluators.
• Exact Match: deterministic comparison of your system output against a reference label.

Agents

• Evaluating an Agent's intermediate steps: compare the sequence of actions taken by an agent to an expected trajectory to grade effective tool use.
• Tool Selection: Evaluate the precision of selected tools. Include an automated prompt writer to improve the tool descriptions based on failure cases.

Multimodel

• Evaluating Multimodal Models: benchmark a multimodal image classification chain

Fundamentals

• Evaluating Existing Runs: add ai-assisted feedback and evaluation metrics to existing run traces.
• Naming Test Projects: manually name your tests with run_on_dataset(..., project_name='my-project-name')
• How to download feedback and examples from a test project: export the predictions, evaluation results, and other information to programmatically add to your reports.

TypeScript / JavaScript Testing Examples

Incorporate LangSmith into your TS/JS testing and evaluation workflow:

• Vision-based Evals in JavaScript: evaluate AI-generated UIs using GPT-4V

We are working to add more JS examples soon. In the meantime, check out the JS eval quickstart the following guides:

• JS LangSmith walkthrough
• Evaluation quickstart

Using Feedback

Harness user feedback, "ai-assisted" feedback, and other signals to improve, monitor, and personalize your applications. Feedback can be user-generated or "automated" using functions or even calls to an LLM:

• Streamlit Chat App: a minimal chat app that captures user feedback and shares traces of the chat application.
  • The vanilla_chain.py contains an LLMChain that powers the chat application.
  • The expression_chain.py contains an equivalent chat chain defined exclusively with LangChain expressions.
• Next.js Chat App: explore a simple TypeScript chat app demonstrating tracing and feedback capture.
  • You can check out a deployed demo version here.
• Building an Algorithmic Feedback Pipeline: automate feedback metrics for advanced monitoring and performance tuning. This lets you evaluate production runs as a batch job.
• Real-time Automated Feedback: automatically generate feedback metrics for every run using an async callback. This lets you evaluate production runs in real-time.
• Real-time RAG Chat Bot Evaluation: This Streamlit walkthrough showcases an advanced application of the concepts from the Real-time Automated Feedback tutorial. It demonstrates how to automatically check for hallucinations in your RAG chat bot responses against the retrieved documents. For more information on RAG, check out the LangChain docs.
• LangChain Agents with LangSmith instrument a LangChain web-search agent with tracing and human feedback.

Optimization

Use LangSmith to help optimize your LLM systems, so they can continuously learn and improve.

• Prompt Bootstrapping: Optimize your prompt over a set of examples by incorporating human feedback and an LLM prompt optimizer.
• Online Few-shot Examples Configure online evaluators to add good examples to a dataset. Review, then use them as few-shot examples to boost performance.

Exporting data for fine-tuning

Fine-tune an LLM on collected run data using these recipes:

• OpenAI Fine-Tuning: list LLM runs and convert them to OpenAI's fine-tuning format efficiently.
• Lilac Dataset Curation: further curate your LangSmith datasets using Lilac to detect near-duplicates, check for PII, and more.

Exploratory Data Analysis

Turn your trace data into actionable insights:

• Exporting LLM Runs and Feedback: extract and interpret LangSmith LLM run data, making them ready for various analytical platforms.
• Lilac: enrich datasets using the open-source analytics tool, Lilac, to better label and organize your data.