About classifier
softexplorer opened this issue · comments
Where do you use the classifier in the code of predicting? How the classifer affect output "dev_output_**.json"?
I didn't understand your questions. Could you explain it more details?
The prediction can be got by the function predict_squad(model, dev_data, v2_on=args.v2_on).
I can't find where the result of classifier affect prediction.In function model.predict, if s_idx == len(spans[i]) - 1: answer = '', but the result of classifier 'lab' don't decide if answer = ‘’.Where do the result of classifier decide if answer = ''.
If the index of answer points the last dumpy token, it means that it is an empty answer as attached line.
https://github.com/kevinduh/san_mrc/blob/master/src/model.py#L141
To utilize results of classifier, you may need to select a threshold based on dev by calling the official evaluation script. As the attached example, if we select the threshold 0.2824*, it gives best F1 70.1.
Hope this answers your questions.
OrderedDict([('exact', 66.29327044554873), ('f1', 69.08254766223166), ('total', 11873), ('HasAns_exact', 61.94331983805668), ('HasAns_f1', 67.52987321080874), ('HasAns_total', 5928), ('NoAns_exact', 70.63078216989067), ('NoAns_f1', 70.63078216989067), ('NoAns_total', 5945), ('best_exact', 67.48926134927987), ('best_exact_thresh', 0.26524031162261963), ('best_f1', 70.0682365711497), ('best_f1_thresh', 0.2824656665325165)])