Type Error problem,can you help me to solve the following problems? the runnning logs are as follow:
lizxko2009 opened this issue · comments
(updatenet) cxhpc@cxhpc:/media/cxhpc/Samsung_T5/snlt-main$ python run_experiment.py
Some weights of the model checkpoint at /media/cxhpc/Samsung_T5/snlt-main/checkpoints/bert-base-chinese/ were not used when initializing BertModel: ['cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.weight']
- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
21-06-30 19:40:22 - INFO : input_pipeline.paired_patches_dataset - BUILDING PAIRED PATCHES DATASET...
Reading MSCOCO Images: 100%|█████████| 117266/117266 [00:05<00:00, 20493.08it/s]
21-06-30 19:40:31 - INFO : input_pipeline.paired_patches_dataset - SIZE OF THE DATASET IS: 859999
21-06-30 19:40:31 - INFO : input_pipeline.paired_patches_dataset - SHUFFLING THE ENTIRE TRAINING SET. MAY TAKE A WHILE...
21-06-30 19:40:33 - INFO : input_pipeline.paired_patches_dataset - SHUFFLING COMPLETED.
Reading MSCOCO Images: 100%|█████████| 117266/117266 [00:05<00:00, 20639.32it/s]
21-06-30 19:40:42 - INFO : input_pipeline.paired_patches_dataset - SIZE OF THE TEST DATASET IS: 859999
21-06-30 19:40:42 - INFO : input_pipeline.paired_patches_dataset - SHUFFLING THE ENTIRE TEST SET. MAY TAKE A WHILE...
21-06-30 19:40:42 - WARNING : tracker.trainer - NOT LOADING ANY WEIGHTS. INITIALIZED FROM SCRATCH.
21-06-30 19:40:42 - INFO : tracker.trainer - TRAINING PREPARATION DONE.
21-06-30 19:40:42 - INFO : tracker.trainer - TRAINING BACKBONE.
21-06-30 19:40:42 - INFO : tracker.trainer - EPOCH: 0
21-06-30 19:40:42 - INFO : tqdm - Training Epoch 0: 0%| | 0/4688 [00:00<?, ?it/s]
Process SpawnProcess-1:
Traceback (most recent call last):
File "/home/cxhpc/anaconda3/envs/updatenet/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
fn(i, *args)
File "/media/cxhpc/Samsung_T5/snlt-main/run_experiment.py", line 50, in run_training
trainer.train_tracker_on_datasets(tracker, train_dataset, test_dataset, dist_rank)
File "/media/cxhpc/Samsung_T5/snlt-main/tracker/trainer.py", line 33, in train_tracker_on_datasets
rank)
File "/media/cxhpc/Samsung_T5/snlt-main/tracker/trainer.py", line 114, in _train_one_epoch
for batch in train_dataloader:
File "/home/cxhpc/anaconda3/envs/updatenet/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 435, in next
data = self._next_data()
File "/home/cxhpc/anaconda3/envs/updatenet/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1085, in _next_data
return self._process_data(data)
File "/home/cxhpc/anaconda3/envs/updatenet/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1111, in _process_data
data.reraise()
File "/home/cxhpc/anaconda3/envs/updatenet/lib/python3.7/site-packages/torch/_utils.py", line 428, in reraise
raise self.exc_type(msg)
AttributeError: Caught AttributeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/cxhpc/anaconda3/envs/updatenet/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 198, in _worker_loop
data = fetcher.fetch(index)
File "/home/cxhpc/anaconda3/envs/updatenet/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/cxhpc/anaconda3/envs/updatenet/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/media/cxhpc/Samsung_T5/snlt-main/input_pipeline/paired_patches_dataset.py", line 108, in getitem
search_box = self._get_bbox(search_image, xyxybox, self.exemplar_size)
File "/media/cxhpc/Samsung_T5/snlt-main/input_pipeline/paired_patches_dataset.py", line 81, in _get_bbox
imh, imw = image.shape[:2]
AttributeError: 'NoneType' object has no attribute 'shape'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/cxhpc/anaconda3/envs/updatenet/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/home/cxhpc/anaconda3/envs/updatenet/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/home/cxhpc/anaconda3/envs/updatenet/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 26, in _wrap
sys.exit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/cxhpc/anaconda3/envs/updatenet/lib/python3.7/multiprocessing/process.py", line 300, in _bootstrap
util._exit_function()
File "/home/cxhpc/anaconda3/envs/updatenet/lib/python3.7/multiprocessing/util.py", line 357, in _exit_function
p.join()
File "/home/cxhpc/anaconda3/envs/updatenet/lib/python3.7/multiprocessing/process.py", line 140, in join
res = self._popen.wait(timeout)
File "/home/cxhpc/anaconda3/envs/updatenet/lib/python3.7/multiprocessing/popen_fork.py", line 48, in wait
return self.poll(os.WNOHANG if timeout == 0.0 else 0)
File "/home/cxhpc/anaconda3/envs/updatenet/lib/python3.7/multiprocessing/popen_fork.py", line 28, in poll
pid, sts = os.waitpid(self.pid, flag)
File "/home/cxhpc/anaconda3/envs/updatenet/lib/python3.7/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 27968) is killed by signal: Terminated.
21-06-30 19:41:25 - INFO : tqdm -
Traceback (most recent call last):
File "run_experiment.py", line 103, in
mp.spawn(run_training, args=(cfg,), nprocs=cfg.NUM_GPU_PER_MACHINE, join=True)
File "/home/cxhpc/anaconda3/envs/updatenet/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 199, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/home/cxhpc/anaconda3/envs/updatenet/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 157, in start_processes
while not context.join():
File "/home/cxhpc/anaconda3/envs/updatenet/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 118, in join
raise Exception(msg)
Exception:
-- Process 0 terminated with the following error:
Traceback (most recent call last):
File "/home/cxhpc/anaconda3/envs/updatenet/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
fn(i, *args)
File "/media/cxhpc/Samsung_T5/snlt-main/run_experiment.py", line 50, in run_training
trainer.train_tracker_on_datasets(tracker, train_dataset, test_dataset, dist_rank)
File "/media/cxhpc/Samsung_T5/snlt-main/tracker/trainer.py", line 33, in train_tracker_on_datasets
rank)
File "/media/cxhpc/Samsung_T5/snlt-main/tracker/trainer.py", line 114, in _train_one_epoch
for batch in train_dataloader:
File "/home/cxhpc/anaconda3/envs/updatenet/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 435, in next
data = self._next_data()
File "/home/cxhpc/anaconda3/envs/updatenet/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1085, in _next_data
return self._process_data(data)
File "/home/cxhpc/anaconda3/envs/updatenet/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1111, in _process_data
data.reraise()
File "/home/cxhpc/anaconda3/envs/updatenet/lib/python3.7/site-packages/torch/_utils.py", line 428, in reraise
raise self.exc_type(msg)
AttributeError: Caught AttributeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/cxhpc/anaconda3/envs/updatenet/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 198, in _worker_loop
data = fetcher.fetch(index)
File "/home/cxhpc/anaconda3/envs/updatenet/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/cxhpc/anaconda3/envs/updatenet/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/media/cxhpc/Samsung_T5/snlt-main/input_pipeline/paired_patches_dataset.py", line 108, in getitem
search_box = self._get_bbox(search_image, xyxybox, self.exemplar_size)
File "/media/cxhpc/Samsung_T5/snlt-main/input_pipeline/paired_patches_dataset.py", line 81, in _get_bbox
imh, imw = image.shape[:2]
AttributeError: 'NoneType' object has no attribute 'shape'
It seems that the image read was None. You can double check if the path to the image is correct.