Mismatch result in test_resnet
KumoLiu opened this issue · comments
======================================================================
FAIL: test_resnet_pretrained_00 (tests.test_resnet.TestResNet)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
return func(*(a + p.args), **p.kwargs, **kw)
File "/opt/monai/tests/test_resnet.py", line 314, in test_resnet_pretrained
equal_state_dict(pretrained_net.state_dict(), medicalnet_state_dict)
File "/opt/monai/tests/utils.py", line 826, in equal_state_dict
assert_allclose(val_st_1, val_st_2)
File "/opt/monai/tests/utils.py", line 135, in assert_allclose
np.testing.assert_allclose(actual, desired, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/numpy/testing/_private/utils.py", line 1592, in assert_allclose
assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
File "/usr/lib/python3.10/contextlib.py", line 79, in inner
return func(*args, **kwds)
File "/usr/local/lib/python3.10/dist-packages/numpy/testing/_private/utils.py", line 862, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=1e-07, atol=0
Mismatched elements: 64 / 64 (100%)
Max absolute difference: 0.47085252
Max relative difference: 0.48536402
x: array([0.914212, 0.99489 , 1.044543, 0.977004, 0.932124, 0.964176,
0.954314, 0.998955, 0.95196 , 0.985342, 1.054769, 1.026216,
0.967691, 0.967283, 0.980336, 0.892576, 0.880693, 0.971825,...
y: array([0.930553, 0.940617, 1.000052, 1.042304, 0.958182, 0.895201,
0.882125, 0.950106, 0.880216, 0.926166, 0.940082, 0.938241,
0.925333, 1.023938, 0.890228, 0.897204, 0.930041, 0.888185,...
======================================================================
FAIL: test_resnet_pretrained_01 (tests.test_resnet.TestResNet)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
return func(*(a + p.args), **p.kwargs, **kw)
File "/opt/monai/tests/test_resnet.py", line 314, in test_resnet_pretrained
equal_state_dict(pretrained_net.state_dict(), medicalnet_state_dict)
File "/opt/monai/tests/utils.py", line 826, in equal_state_dict
assert_allclose(val_st_1, val_st_2)
File "/opt/monai/tests/utils.py", line 135, in assert_allclose
np.testing.assert_allclose(actual, desired, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/numpy/testing/_private/utils.py", line 1592, in assert_allclose
assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
File "/usr/lib/python3.10/contextlib.py", line 79, in inner
return func(*args, **kwds)
File "/usr/local/lib/python3.10/dist-packages/numpy/testing/_private/utils.py", line 862, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=1e-07, atol=0
Mismatched elements: 64 / 64 (100%)
Max absolute difference: 0.10385573
Max relative difference: 0.12188552
x: array([0.964119, 0.926743, 0.875544, 0.943817, 0.873043, 0.953679,
0.92949 , 0.946948, 0.931833, 0.868956, 0.924017, 0.963394,
0.946468, 0.956957, 1.024469, 0.922758, 0.894483, 0.955731,...
y: array([0.900147, 0.918002, 0.918516, 0.928497, 0.953533, 0.915175,
0.947617, 0.995937, 0.901341, 0.907229, 0.907972, 0.918252,
0.981896, 0.912952, 0.961669, 0.9233 , 0.858861, 0.851897,...
======================================================================
FAIL: test_resnet_pretrained_02 (tests.test_resnet.TestResNet)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
return func(*(a + p.args), **p.kwargs, **kw)
File "/opt/monai/tests/test_resnet.py", line 314, in test_resnet_pretrained
equal_state_dict(pretrained_net.state_dict(), medicalnet_state_dict)
File "/opt/monai/tests/utils.py", line 826, in equal_state_dict
assert_allclose(val_st_1, val_st_2)
File "/opt/monai/tests/utils.py", line 135, in assert_allclose
np.testing.assert_allclose(actual, desired, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/numpy/testing/_private/utils.py", line 1592, in assert_allclose
assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
File "/usr/lib/python3.10/contextlib.py", line 79, in inner
return func(*args, **kwds)
File "/usr/local/lib/python3.10/dist-packages/numpy/testing/_private/utils.py", line 862, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=1e-07, atol=0
Mismatched elements: 64 / 64 (100%)
Max absolute difference: 0.1821745
Max relative difference: 0.1739692
x: array([0.945007, 1.016222, 1.021641, 0.957939, 0.905201, 0.955808,
0.970912, 0.935607, 0.914829, 0.91266 , 0.87098 , 0.951179,
1.010709, 0.955448, 0.932211, 0.97651 , 0.969711, 0.953455,...
y: array([1.039345, 0.94142 , 0.95192 , 0.922544, 0.933786, 0.92952 ,
1.016223, 0.950827, 0.979054, 0.97786 , 0.953783, 0.934071,
0.939528, 0.996561, 0.968768, 0.969426, 0.969712, 1.003731,...
----------------------------------------------------------------------
Should I revert the commit from that PR?
Should I revert the commit from that PR?
Trying debug the issue. Looks like the bn weight is not updated correctly. If you could help take a look would be better. Thanks.
Looks like line 97 in monai/networks/nets/resnet.py is the culprit. norm_layer = get_norm_layer(name=norm, spatial_dims=spatial_dims, channels=planes). This norm_layer object is then used in 2 layers instead of creating a new norm for the second layer. Tests pass locally after changing that.