Label propgation problem
sunwoo76 opened this issue · comments
First of all, thanks for your great work!
When I do label propagation, this error is happen in the test.py file.
(I followed your 'Evaluation: Label Propagation' tab. in README)
******* Vid 0 TOOK 63.87427091598511 *******
******* Vid 1 (70 frames) *******
computed features 0.48213911056518555
Killed
Why does the process is killed after only processing video 0 ?
How can I solve this problem?
Hi @sunshower76, thanks for your interest!
This looks like it might be an out-of-memory problem. Could you verify that this is the case? In the evaluation script, we precompute all the features for each video.
Thanks for your comment.
I think this is not an out-of-memory problem.
Your code works well now. but, I didn't find the reason that this error caused.
I modified some code that is related to 'npy' path anddavis_valist txt.
thanks :)
@ajabri and Could I give you another question?
If the video or a clip consists of frames that have a length of 69, the code appends additional the first frames 20 times. (this option is --videoLen=20 in evaluation time.)
Why is this option needed?
Why are the additional first frames appended 20 times?
[0.jpg, 0.jpg, 0.jpg, ....0.jpg,1.jpg,2.jpg, 69.jpg]
thank you!
This is done just to make the implementation of using sliding window for temporal context more straightforward. This way, we always use a sliding window of the last L context frames.
Dear @ajabri @sunwoo76 , I also encountered similar autokill problem with provided code (I set up the code on a workstation with 11G 2020 Ti). It seems that it is not related to out-of-memory. Do you have ideas or suggestions? Thanks in advance!
- if I change the args.workers to 0 or 1, it will report
******* Vid 0 (89 frames) *******
computed features 1.028991937637329
Killed
- if args.workers is set to more than 2
Traceback (most recent call last):
File "~/miniconda3/envs/dm4vos/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 872, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "~/miniconda3/envs/dm4vos/lib/python3.7/queue.py", line 179, in get
self.not_empty.wait(remaining)
File "~/miniconda3/envs/dm4vos/lib/python3.7/threading.py", line 300, in wait
gotit = waiter.acquire(True, timeout)
File "~/miniconda3/envs/dm4vos/lib/python3.7/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 953789) is killed by signal: Killed.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "test.py", line 209, in <module>
main(args, vis)
File "test.py", line 58, in main
test_loss = test(val_loader, model, args)
File "test.py", line 65, in test
for vid_idx, (imgs, imgs_orig, lbls, lbls_orig, lbl_map, meta) in enumerate(loader):
File "~/miniconda3/envs/dm4vos/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 435, in __next__
data = self._next_data()
File "~/miniconda3/envs/dm4vos/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1068, in _next_data
idx, data = self._get_data()
File "~/miniconda3/envs/dm4vos/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1024, in _get_data
success, data = self._try_get_data()
File "~/miniconda3/envs/dm4vos/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 885, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
RuntimeError: DataLoader worker (pid(s) 953789) exited unexpectedly
> ~/miniconda3/envs/dm4vos/lib/python3.7/site-packages/torch/utils/data/dataloader.py(885)_try_get_data()
-> raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
Best.
Yujie