π llmonitor
Open-source monitoring & observability for AI apps and agents
Features
LLMonitor helps AI devs monitor their apps in production, with features such as:
- π΅ Cost, token & latency analytics
- π Log & edit prompts
- π Trace agents/chains to debug easily
- πͺ Track users
- π·οΈ Label and export fine-tuning datasets
- π²οΈ Collect feedback from users
- π§ͺ Unit tests & prompt evaluations (soon)
It also designed to be:
- π€ Usable with any model, not just OpenAI
- π¦ Easy to integrate (2 minutes)
- π§βπ» Simple to self-host (deploy to Vercel & Supabase)
Demo
demo720.mp4
βοΈ Integration
Modules available for:
LLMonitor natively supports:
- LangChain (JS & Python)
- OpenAI module
- LiteLLM
Additionally you can use it with any framework by wrapping the relevant methods.
π Documentation
Full documentation is available on the website.
βοΈ Hosted version
We offer a hosted version with a free plan of up to 1k requests / days.
With the hosted version:
- π· don't worry about devops or managing updates
- π get priority 1:1 support with our team
- πͺπΊ your data is stored safely in Europe
π Support
Need help or have questions? Chat with us on Discord or email one of the founders: vince [at] llmonitor.com. We're here to support you every step of the way.
π¨βπ©βπ§ Team
License
This project is licensed under the Apache 2.0 License.