FileNotFoundError: [Errno 2] No such file or directory: '/tmp/$WORK/tempbs_7o9oj'
shuaikangma opened this issue · comments
**I begin to train on my own data, but I get an error when it evals for the first time. The log shows as follow: **
Epoch: [11] [0/8] eta: 0:00:34 loss: 0.0000 (0.0000) learning_rate: 0.0008 (0.0008) time: 4.2506 data: 2.3495 max mem: 9466
Epoch: [11] [7/8] eta: 0:00:00 loss: 0.0000 (0.0000) learning_rate: 0.0008 (0.0008) time: 0.9943 data: 0.2958 max mem: 9491
Epoch: [11] Total time: 0:00:08 (1.0115 s / it)
Epoch: [12] [0/8] eta: 0:00:27 loss: 0.0000 (0.0000) learning_rate: 0.0008 (0.0008) time: 3.4646 data: 2.7603 max mem: 9492
Epoch: [12] [7/8] eta: 0:00:00 loss: 0.0000 (0.0000) learning_rate: 0.0008 (0.0008) time: 0.8330 data: 0.3464 max mem: 9492
Epoch: [12] Total time: 0:00:06 (0.8537 s / it)
Eval: [ 0/58] eta: 0:01:40 time: 1.7340 data: 1.3048 max mem: 10891
Eval: [50/58] eta: 0:00:01 time: 0.1124 data: 0.0121 max mem: 16814
Eval: [57/58] eta: 0:00:00 time: 0.1047 data: 0.0120 max mem: 16814
_Eval: Total time: 0:00:08 (0.1505 s / it)
Traceback (most recent call last):
File "/home/qiuzheng/.conda/envs/Segmenter/lib/python3.8/runpy.py", line 192, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/qiuzheng/.conda/envs/Segmenter/lib/python3.8/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/qiuzheng/segmenter/segm/train.py", line 304, in
main()
File "/home/qiuzheng/.conda/envs/Segmenter/lib/python3.8/site-packages/click/core.py", line 1128, in call
return self.main(*args, **kwargs)
File "/home/qiuzheng/.conda/envs/Segmenter/lib/python3.8/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/home/qiuzheng/.conda/envs/Segmenter/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/qiuzheng/.conda/envs/Segmenter/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/home/qiuzheng/segmenter/segm/train.py", line 266, in main
eval_logger = evaluate(
File "/home/qiuzheng/.conda/envs/Segmenter/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
return func(*args, **kwargs)
File "/home/qiuzheng/segmenter/segm/engine.py", line 104, in evaluate
val_seg_pred = gather_data(val_seg_pred)
File "/home/qiuzheng/segmenter/segm/metrics.py", line 60, in gather_data
tmpdir = tempfile.mkdtemp(prefix=tmpprefix)
File "/home/qiuzheng/.conda/envs/Segmenter/lib/python3.8/tempfile.py", line 359, in mkdtemp
os.mkdir(file, 0o700)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/$WORK/tempbs_7o9oj'
Just wanted to write and say that I've had the same issue.
os.mkdir(file, 0o700)改成os.makedirs(file,0o700)
Hi,
Do you have a locally modified version of the function gather_data in segm/metrics.py ?
By default this function creates a temporary directory in "$WORK/temp" as specified in this line https://github.com/rstrudel/segmenter/blob/master/segm/metrics.py#L53 .
I can see in your case it's trying to create a folder in : /tmp/$WORK/tempbs_7o9oj'
So it seems the environment variable $WORK is not replaced by its actual value and it is different from the default temp value.
Maybe you are passing your tmp_dir or modified it a bit. If you use environment variables, make sure to sure os.path.expandvars to make the path valid.
It seems os.path.expandvars was missing in the case where the user pass its own tmp_dir, I just pushed.
My workaround was just defining $WORK in my .bashrc to the segmenter folder then creating a temp folder there.
Ok thank you very much for the pointer. I will replace the WORK by DATASET to avoid bugs