Evaluation time-consuming on FIQA

[Yaxin Du]

Hi,

I hope this message finds you well. First and foremost, I would like to express my gratitude for the incredible work you have put into this project; it has been instrumental in my work.

I am reaching out to seek some guidance and insights regarding the evaluation time of the model across different test sets. In my current setup, I am observing that the evaluation phase is significantly time-consuming, for example on FIQA roughly taking around two to three hours to complete even in A6000 with batch size 64 for LLAMA. This duration seems to persist across various test sets, which has brought me to seek your expertise.

I am wondering if there might be any specific recommendations or strategies that could potentially help in accelerating the evaluation process.

Here are a few questions I have in mind:

Are there any known bottlenecks in the evaluation process for FIQA that I should be aware of?
Could you please suggest any best practices or settings that could help in reducing the evaluation time?
Is there any parallelization or optimization technique available that is recommended for speeding up the evaluation?
I am more than willing to provide additional information or clarify any aspects if needed. My main goal is to ensure that I am utilizing the tool to its fullest potential and in the most efficient manner.

Thank you very much for taking the time to read my inquiry. I am looking forward to any advice or suggestions you might have.