points and features are mismatch in PointnetSAModuleMatch module

[encore-zhou]

Hi, I find that there may be some mistakes in PointnetSAModuleMatch module.

In file proposal_module_refine.py:
line 328:
surface_xyz, surface_features, _ = self.match_surface_center(torch.cat((obj_surface_center, surface_center_pred), dim=1), torch.cat((obj_surface_feature, surface_center_feature_pred), dim=2))

In line 119: match_surface_center is defined as:

### surface center matching
       self.match_surface_center = PointnetSAModuleMatch( 
            npoint=self.num_proposal*6,
            radius=0.5,
            nsample=32,
            mlp=[128+6, 128, 64, 32],
            use_xyz=True,
            normalize_xyz=True
        )

As for torch.cat((obj_surface_center, surface_center_pred), dim=1) and torch.cat((obj_surface_feature, surface_center_feature_pred), dim=2), we can get a set of points and their correspondence features.

Then, I go further into the source code of PointnetSAModuleMatch module:
pointnet2_modules.py: Line 467:

        new_xyz = xyz[:,:self.npoint,:].contiguous()
        target_xyz = xyz[:,self.npoint:,:].contiguous()
        
        if not self.ret_unique_cnt:
            grouped_features, grouped_xyz = self.grouper(
                target_xyz, new_xyz, features
            )  # (B, C, npoint, nsample)
        else:
            grouped_features, grouped_xyz, unique_cnt = self.grouper(
                target_xyz, new_xyz, features
            )  # (B, C, npoint, nsample), (B,3,npoint,nsample), (B,npoint)

Here, I think the difference between PointnetSAModuleMatch and original PointnetSAModule is that user can specify new_xyz in PointnetSAModuleMatch module.
However, I found some problems in Line 472. target_xyz and features are two parameters parsed to function self.grouper, but they are mismatch in these case. Problem will happen while grouping the neighbour features, because the features of target_xyz is begin from features[:,:,self.npoint:]. I think we should correcte this code with

grouped_features, grouped_xyz, unique_cnt = self.grouper(
                target_xyz, new_xyz, features[:,:,self.npoint:].contiguous()
            )

[zaiweizhang]

I just checked. I agree with you. Let me change the code and test it. Thank you for pointing this out!

[zaiweizhang]

Thank you for your suggestion! We have fixed the issue, and it offers a marginal improvement on ScanNet, and does nothing on SUNRGB-D. We have updated the code. Thank you, again!

[encore-zhou]

Hi, Thanks for your reply.
I have other questions as below.

1. I fixed the problem as mentioned before, but I only get a small improvement on ScanNet. Could you share the performance and the training configs?
2. The outputs of PrimitiveModule consists of three parts(offset, semantic scores and size residuals). The offset and semantic is well explained in the paper. But I cannot find what "size residuals" means. And I found that the size residuals of different primitives are not used during inference time.
   For example, in file proposal_module_surface.py: Line 28:

def decode_scores(net, end_points, num_class, mode=''):
    net_transposed = net.transpose(2,1) # (batch_size, 1024, ..)
    batch_size = net_transposed.shape[0]
    num_proposal = net_transposed.shape[1]
    
    base_xyz = end_points['aggregated_vote_xyz'+mode] # (batch_size, num_proposal, 3)
    center = base_xyz + net_transposed[:,:,0:3] # (batch_size, num_proposal, 3)
    end_points['center'+mode] = center

    if mode == '_z':
        end_points['size_residuals'+mode] = net_transposed[:,:,3:5]
        sem_cls_scores = net_transposed[:,:,5:] # Bxnum_proposalx10
        end_points['sem_cls_scores'+mode] = sem_cls_scores
    elif mode == '_xy':
        end_points['size_residuals'+mode] = net_transposed[:,:,3:4]
        sem_cls_scores = net_transposed[:,:,4:] # Bxnum_proposalx10
        end_points['sem_cls_scores'+mode] = sem_cls_scores
    else:
        sem_cls_scores = net_transposed[:,:,3:] # Bxnum_proposalx10
        end_points['sem_cls_scores'+mode] = sem_cls_scores
    return center, end_points

Size residuals are decoded from the outputs of PrimitiveModule.
Thank You!

[zaiweizhang]

For question 1, when I fixed the bug, it just gives me marginal improvements, which mean small incremental improvements, in ScanNet.

For question 2,
Yes, Size residuals means the length of an edge or size of a surface. It's not used in inference. For now, we just use it as a additional supervision and hope it can offer some improvements through multi-task. However, if you remove it, I doubt that it will have any impact. It's not an important factor.

[zaiweizhang]

Btw, I was just curious and I looked at your github. I found that you were trying to add the primitive part to Open-mmlab, right? Did you get any improvements by adding the primitive part? I am just curious. If it's confidential, feel free to not share it.