Why Do we Need Pre-Computed for Test Images

[TouqeerAhmad]

Hello @Brummi, I was trying to run inference using scripts/images_decomposition_co3d.py, however, it looks like the code always relies on precomputed stuff e.g., depth, albedo, normal map etc. For example when running for the category = 'bench', the test_path_precompute is always set to 'datasets/co3d/extracted_bench/precomputed/val' and inspecting the dictionary (data_dict) shows all the tensors are already set essentially to non-zero values:

print('data_dict.keys(): ', data_dict.keys())
print(data_dict['input_im'].shape)
print(data_dict['recon_albedo'].shape)
print(data_dict['recon_depth'].shape)
print(data_dict['recon_normal'].shape)
print(data_dict['recon_normal'][0,0,:10,:10])

Shouldn't the test image be run without relying on any pre-computed inputs and just the single RGB image? as that is the perception I had reading the paper. All these tensors should be initialized to zero except the input_im. May be having a single minimum inference example can help here: loading the model and running inference for a sample face/object image.

[Felix Wimbauer]

Hi,
in this codebase "precomputed" and "extracted" correspond to the ground truth. You can set the corresponding path to None (as for the photos category, where I run the model on pictures I took with my phone) and they will not be loaded. The model does not rely on them during inference.

As the loss is calculated as part of the forward step, we pass this data with data datadict to the model. Within the model class at line 375, all these dict items get the prefix "lr_" for low resolution or coarse. In the visualization script, we load this data to show the comparison. They get plotted / saved from line 248 in the decomposition script onwards.

If you want to run the method on custom data, you actually just do it the same way I did with the photos. I think you should put the image under datasets/photos/imgs_cropped/val.

[TouqeerAhmad]

Thanks you for the detailed explanation! That path suggestion worked well for me.

[Felix Wimbauer]

Looks very cool!