_IncompatibleKeys failure loading weights.

[Raul]

Hi,

Thank you for sharing the code and weights. I am trying to load the surface normal estimator using the config and weights linked in https://github.com/SysCV/idisc?tab=readme-ov-file#normals:

• https://github.com/SysCV/idisc/blob/main/configs/nyunorm/nyunorm_swinl.json
• https://dl.cv.ethz.ch/idisc/checkpoints/nyunormals_swinlarge.pt
  And there is an error loading the weights:

import json
from idisc.models.idisc import IDisc

NORMALS_CONFIG_FILE = "models/nyunorm_swinl.json"
NORMALS_MODEL = "models/nyunormals_swinlarge.pt"
with open(NORMALS_CONFIG_FILE, "r") as f:
    config = json.load(f)

model = IDisc.build(config=config)
model.load_pretrained(NORMALS_MODEL)

-> Encoder is pretrained from: https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_large_patch4_window7_224_22k.pth Loading pretrained info: _IncompatibleKeys(missing_keys=['norm0.weight', 'norm0.bias', 'norm1.weight', 'norm1.bias', 'norm2.weight', 'norm2.bias', 'norm3.weight', 'norm3.bias'], unexpected_keys=['norm.weight', 'norm.bias', 'head.weight', 'head.bias', 'layers.0.blocks.1.attn_mask', 'layers.1.blocks.1.attn_mask', 'layers.2.blocks.1.attn_mask', 'layers.2.blocks.3.attn_mask', 'layers.2.blocks.5.attn_mask', 'layers.2.blocks.7.attn_mask', 'layers.2.blocks.9.attn_mask', 'layers.2.blocks.11.attn_mask', 'layers.2.blocks.13.attn_mask', 'layers.2.blocks.15.attn_mask', 'layers.2.blocks.17.attn_mask'])

It also happens with Swin-Large for NYU2 Depth Estimation.
Am I missing something?

Thank you.

[Luigi Piccinelli]

Hi,

The warning you see is expected, the code loads the SWin weights first (giving this warning) and then loads iDisc weights, which should not trigger any warning/error, and are the ones actually used.

FYI, the part about normalizations (norm) is due to dropping the classifier norm and adding norms for every output layer (these are not in the original SWin, but iDisc models have them). While the attention masks (attn_mask) are re-computed and are just static buffers.