GPT Monitor is a project that monitors OpenAI's GPT models by measuring certain metrics of their responses to a set of prompts. The project aims to both track the changes and progress of the models over time by analyzing various metrics of their responses, and to provide a public benchmark of performance over time.
Other metrics include ms per token, time to first byte, and time to last byte.
Note: this is a small fire-and-forget pet project. It is not intended to be used for anything serious.
The following improvements would make this better, but cost more:
- Run multiple requests each time and use the mean
- Run requests more frequently (currently at 15 minutes)
Also
- Providing a granular last 24 hours and a less granular mean of daily values
- More metrics, such as tokens per minute per sequence length (another project perhaps)
Feel free to submit issues or pull requests if you'd like to contribute to the project or have any suggestions for improvements.
// Copyright (C) 2023 Adam K Dean <adamkdean@googlemail.com>
// Use of this source code is governed by the GPL-3.0
// license that can be found in the LICENSE file.