How "SOTA" should results be?
kevin-parnow opened this issue · comments
Hey, I'm wondering about the guidelines for whether a paper should be added. Obviously, the current best-performing model for a given metric should be added, but what if there's a similar but slightly lower-performing model released at the same conference/in the same couple months/etc.?
Not sure if this is meant to be more of an overall leaderboard for all-time bests or a timeline of the bests as they happened.
Hi,
In general, I think it's good to err on the side of completeness and to highlight as many competitive models as we can. Some models might perform better but not have any code available, so it's good to give people a choice. So in general, if you are aware of a model that achieves a similar or slightly worse performance as the best model and has already been published somewhere, feel free to add it.
Thank you, glad to clarify this.