question about calculating NDCG score

[Eku0194]

Your code shows :
class NDCG(Metric):
def init(self, k=10, gain_type='exp2'):
super(NDCG, self).init()
self.k = k
self.gain_type = gain_type
self._dcg = DCG(k=k, gain_type=gain_type)
self._ideals = {}

def evaluate(self, qid, targets):
    return (self._dcg.evaluate(qid, targets) /
            max(_EPS, self._get_ideal(qid, targets)))

Question:

1. Is evaluate method used to calculate the NDCG score of the model?
   If yes, then what is the format of targets ? Is it the array of labels for each query-document pair?
   If No, then how do we calculate NDCG score of the selected model?

[Jerry Ma]

Hello! targets is an ordered list of relevance scores, where targets[0] is the top of the result set. And the evaluate() method is used to calculate the NDCG score of each individual result set. You could sum and divide by n to get the average NDCG across the dataset.

[Eku0194]

Thank you, Jerry.
So, the relevance score is the result of the ranking function?
In order to calculate average NDCG for multiple queries at once - do I have to sum all the NDCG score of each individual result set and then divide by n or I need to calculate average NDCG per query_id and then divide it by the total no. of queries?

[Jerry Ma]

There's a one-to-one correspondence between a query and a result set (i.e. if n denotes the number of result sets, then n is equal to the total number of queries).

[Eku0194]

Thank you so much, Jerry!