Preface: https://github.com/sheetalkalburgi/sentence-similarity
Upon performing the evaluation and analyzing the predictions with the client, it was discovered that although the equivalent regulations between FDA and Health Canada were established and the models were predicting accurately, they were not a true match. The main concern were the regulations which are not matched as per the ground truth, but the model is predicting the said regulations to be a match. In other words, false positives being generated form the model is of focus, and it is crucial to minimize the number of false positives being generated. Therefore, the accuracy is no longer the optimizing metric, but precision is.
Analysis on my Medium!