Progress Bar may use more resources than necessary
soldni opened this issue · comments
from @dirkgr:
When I have a console window open that watches dolma tokens, it literally takes 1.5 cores permanently on my Mac.
allenai / dolma
Data and tools for generating and inspecting OLMo pre-training data.