Issue with Irrelevant and Low Novelty Paper Recommendations
Neph0s opened this issue · comments
Description
Hello, I am highly interested in your project but have encountered some issues during execution. I forked the repository and ran the code manually without altering any configuration files, as per the instructions on GitHub. The execution resulted in six paper recommendations. However, upon reviewing the comments of these papers, it was evident that they were irrelevant to the given topic, scoring only 3-4 in both relevance and novelty.
Concern
This outcome is quite puzzling, especially since it contrasts with the high relevance seen in the examples provided in your repository. I am unsure about the cause of this discrepancy and would appreciate any guidance or insights you could offer.
There is randomness in the LM even at temp zero, and at low relevance scores this randomness can be pretty big.
The whole system is tuned for high recall, so it'll err on the side of including weird papers. You can modify the prompt (esp, the instruction tuning category which has high false positives) or adjust the cutoffs in config.ini please report back if you find some magic prompt template that works well.