Giters

voidism / EAR

Code for the ACL 2023 long paper - Expand, Rerank, and Retrieve: Query Reranking for Open-Domain Question Answering

Home Page:https://arxiv.org/abs/2305.17080

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

memory overflow in bm25 retrieval loop when constructing dataset

yjqangiai opened this issue · comments

commented

When I attempted to construct the training data, I encountered a memory overflow (128GB) when performing bm25 retrieval iteratively on CPU. The memory is rapidly increased in the loop proccess. So I would like to ask how to solve it. Thanks.