allenai / dolma

Data and tools for generating and inspecting OLMo pre-training data.

Home Page:https://allenai.github.io/dolma/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Some race condition in url taggers

peterbjorgensen opened this issue · comments

Even with the latest git version some of the URL taggers crash if I run the taggers with multiprocessing. I can't figure out where this race condition happens. If I run the taggers with --processes 1 first and then do --processes 16 it works.

Hi @peterbjorgensen! which taggers are you trying to run? More information for me to reproduce would be great.