https://eugeneyan.com/writing/serendipity-and-accuracy-in-recommender-systems/
utterances-bot opened this issue · comments
Serendipity: Accuracy’s Unpopular Best Friend in Recommenders
What I learned about measuring diversity, novelty, surprise, and serendipity from 10+ papers.
https://eugeneyan.com/writing/serendipity-and-accuracy-in-recommender-systems/
Thank you for this article and having gathered the results of papers. I really appreciate.
Maybe, another reason why relevance is considered as more important could be that the famous Netflix Prize was about improving such a metric.
Thank you very much for this nice summary. Although I just wanted to mention that I wondering if there isn't an error in your second formulation for the novelty:
nov(i)=1-(count(users recommended i)/count(users who have not interacted with i))
Let's say that I recommended an item to all of my users and only half of them interacted with it. In that case nov(i)=1-2=-1.
The range of values for this metric is very weird, something like [-N+1, 1].
Hey Adrien, you're right about the metric being weird—it's likely I wasn't thinking straight when I came up with it.
Excluding the case where no users are recommended the item (i.e., numerator = 0), I was thinking that novelty = 0 indicates lowest novelty (i.e., all users have interacted with the item), while novelty < 0 means there are users who have not interacted with the item. We can then sort by each item's novelty score to re-rank items accordingly (while also considering other metrics such as popularity, sales, etc.)
On hindsight, it might have been better to multiply the existing formula by -1 and perhaps log it.
Thank you very much for the summary on not so common recsys metrics, i kinda curious, in above you mention above the unexpectedness
"Lower cosine similarity indicates higher unexpectedness.", however after i see the formula the similarity factor is being nominator, the lower the similarity the lower unexpectedness ? can i have more clear overview about it ?