mlcommons / modelgauge

Make it easy to automatically and uniformly measure the behavior of many AI Systems.

Home Page:https://mlcommons.org/ai-safety/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Should caching include the API in the key?

brianwgoldman opened this issue · comments

Imagine you have two APIs with exactly the same structure, but different underlying behavior. Maybe something like a "V1" and "V2" where the inputs are the same, but V2 fixes some flaw. In our current way of caching, we would accidentally treat responses to V1 as valid responses to V2. This is because the cache key is the json representation of the native request, and at least some SUTs (AKA Together) don't include the request URL in the request.

It seems pretty unlikely that two APIs would have identical input structure but we should expect them to return different behavior, so this feels like a low priority issue.