Beaker Executor opens a ton of files
dirkgr opened this issue · comments
🐛 Describe the bug
You have to do this in a catwalk context, on commit e7c5d158b9e8f1c925b4894037b5371a1efdeab7.
from tango import StepGraph
sg = StepGraph.from_file("experiments/everything/everything.jsonnet")
from tango import Workspace
ws = Workspace.from_url("beaker://ai2/task-complexity")
s = sg['trained_model_sciq_bert-base-uncased_1337']
r = ws.step_result(s)At this point the Beaker Workspace will open millions of files, quickly exhausting the open file limit. Even when you Ctrl+C the proceedings, background threads keep trying to open files. Your console is now useless.
Versions
asdasdasd
I do this on a Mac. I think Linux has a higher default open file limit, so it takes a lot more to hit the same problem.
I realize (now) that step_result() is not the correct method to call. But the behavior is quite pathological.