AssertionError: wrong patch size

[Xiyi Chen]

Train epoch=0: 0%| | 0/1000000 [00:00<?, ?it/s]
Train epoch=0: 0%| | 0/1000000 [00:05<?, ?it/s]
Train: 0%| | 0/1 [00:07<?, ?it/s]
Traceback (most recent call last):
File "train.py", line 264, in
train_human(opt)
File "train.py", line 154, in train_human
trainer.train()
File "/cluster/scratch/xiychen/ml-neuman/trainers/human_nerf_trainer.py", line 193, in train
self.train_epoch()
File "/cluster/scratch/xiychen/ml-neuman/trainers/human_nerf_trainer.py", line 618, in train_epoch
leave=True,
File "/cluster/scratch/xiychen/miniconda3/envs/neuman_venv/lib/python3.7/site-packages/tqdm/std.py", line 1195, in iter
for obj in iterable:
File "/cluster/scratch/xiychen/miniconda3/envs/neuman_venv/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 530, in next
data = self._next_data()
File "/cluster/scratch/xiychen/miniconda3/envs/neuman_venv/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1224, in _next_data
return self._process_data(data)
File "/cluster/scratch/xiychen/miniconda3/envs/neuman_venv/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1250, in _process_data
data.reraise()
File "/cluster/scratch/xiychen/miniconda3/envs/neuman_venv/lib/python3.7/site-packages/torch/_utils.py", line 457, in reraise
raise exception
AssertionError: Caught AssertionError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/cluster/scratch/xiychen/miniconda3/envs/neuman_venv/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
data = fetcher.fetch(index)
File "/cluster/scratch/xiychen/miniconda3/envs/neuman_venv/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/cluster/scratch/xiychen/miniconda3/envs/neuman_venv/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 49, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/cluster/scratch/xiychen/ml-neuman/datasets/human_rays.py", line 169, in getitem
assert temp[seed[0]:bound[0], seed[1]:bound[1]].shape == (PATCH_SIZE, PATCH_SIZE), 'wrong patch size'
AssertionError: wrong patch size

I got this error when training the human model on one of my custom videos. It happened after loading the near/far cache files. The training for the scene went well, and this didn't happen to my other videos. Any suggestions on what could go wrong?

[Wei Jiang]

Could you print the seed/bound/temp.shape? I think the seed value probably is wrong.

[Xiyi Chen]

@jiangwei221 Thank you for your quick reply, the shapes are (2,) (2,) (715, 1273, 3)
I was wondering if cap.mask should only have one channel?

[Wei Jiang]

It's weird that the seed is not a 2D coordinate, while it should be:

ml-neuman/datasets/human_rays.py

Line 34 in 0149d25

 return np.array([lu_x, lu_y]) 

Can you also print the shape of the cap.mask?
cap.mask is 2D mask with one-channel and is binary:

ml-neuman/data_io/neuman_helper.py

Lines 55 to 65 in 0149d25

 def mask(self): 
     _mask = self.captured_mask.image.copy() 
     if _mask.max() == 255: 
         # Detectron2 mask 
         _mask[_mask == 255] = 1 
         _mask = 1 - _mask 
     else: 
         raise ValueError 
     assert _mask.sum() > 0 
     assert _mask.shape[0:2] == self.pinhole_cam.shape, f'mask does not match with camera model: mask shape: {_mask.shape}, pinhole camera: {self.pinhole_cam}' 
     return _mask 

[Xiyi Chen]

I found that somehow cap.mask has 3 channels: (715, 1273, 3). Does the image reader guarantee it's read as grayscale? I extracted the first channel in the mask function and now it is working correctly

[Xiyi Chen]

I checked imageio.imread and the masks are indeed read as one-channel, not sure how it becomes 3 channels when loaded into cap.mask

[Wei Jiang]

hmmm, not sure why, maybe you can add one more assertion to make sure the number of channels is correct...