memory overflow in bm25 retrieval loop when constructing dataset
yjqangiai opened this issue · comments
When I attempted to construct the training data, I encountered a memory overflow (128GB) when performing bm25 retrieval iteratively on CPU. The memory is rapidly increased in the loop proccess. So I would like to ask how to solve it. Thanks.