Bug in random_channel_mask

[xieydd]

if i set epoch_start_cs=30 and when epoch=0, and epoch_after_cs will be -30.

epoch_delay_early = {0: 0,  # 8
                             1: 1, 2: 1,  # 7
                             3: 2, 4: 2, 5: 2,  # 6
                             6: 3, 7: 3, 8: 3, 9: 3,  # 5
                             10: 4, 11: 4, 12: 4, 13: 4, 14: 4,
                             15: 5, 16: 5, 17: 5, 18: 5, 19: 5, 20: 5,
                             21: 6, 22: 6, 23: 6, 24: 6, 25: 6, 27: 6, 28: 6,
                             29: 6, 30: 6, 31: 6, 32: 6, 33: 6, 34: 6, 35: 6, 36: 7,
                           }
        epoch_delay_late = {0: 0,
                            1: 1,
                            2: 2,
                            3: 3,
                            4: 4, 5: 4,  # warm up epoch: 2 [1.0, 1.2, ... 1.8, 2.0]
                            6: 5, 7: 5, 8: 5,  # warm up epoch: 3 ...
                            9: 6, 10: 6, 11: 6, 12: 6,  # warm up epoch: 4 ...
                            13: 7, 14: 7, 15: 7, 16: 7, 17: 7,  # warm up epoch: 5 [0.4, 0.6, ... 1.8, 2.0]
                            18: 8, 19: 8, 20: 8, 21: 8, 22: 8, 23: 8}  # warm up epoch: 6, after 17, use all scales

        if 0 <= epoch_after_cs <= 23 and self.stage_out_channels[0] >= 64:
            delayed_epoch_after_cs = epoch_delay_late[epoch_after_cs]
        elif 0 <= epoch_after_cs <= 36 and self.stage_out_channels[0] < 64:
            delayed_epoch_after_cs = epoch_delay_early[epoch_after_cs]
        else:
            delayed_epoch_after_cs = epoch_after_cs # delayed_epoch_after_cs = -30

                        channel_scale_start = max(2, 10 - (-30) - 2) # channel_scale_start=38
                        channel_choice = random.randint(channel_scale_start, len(self.candidate_scales) - 1) #random.randint(38, 9) will be error ValueError: empty range for randrange() (38,10, -28)
                        

And why epoch_delay_early doesn`t have the key 26

[Alex]

There is an unit test specifically for verifying whether cs-warm-up working well.

To do so, please change FIX_ARCH to False here
https://github.com/CanyonWind/MXNet-Single-Path-One-Shot-NAS/blob/ce755214e07a38d6d45213d31270b257e004717d/oneshot_nas_network.py#L365

and run python oneshot_nas_network.py

As far as my experiment, CS starts from epoch 30 and gradually grows to all [0.6 - 2.0] choices correctly. The bug cannot be reproduced.

Epoch 0: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
Epoch 1: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
Epoch 2: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
Epoch 3: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
Epoch 4: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
Epoch 5: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
Epoch 6: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
Epoch 7: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
Epoch 8: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
Epoch 9: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
Epoch 10: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
Epoch 11: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
Epoch 12: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
Epoch 13: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
Epoch 14: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
Epoch 15: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
Epoch 16: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
Epoch 17: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
Epoch 18: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
Epoch 19: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
Epoch 20: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
Epoch 21: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
Epoch 22: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
Epoch 23: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
Epoch 24: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
Epoch 25: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
Epoch 26: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
Epoch 27: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
Epoch 28: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
Epoch 29: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
Epoch 30: [9, 9, 9, 9, 8, 9, 8, 8, 9, 9, 8, 8, 9, 8, 9, 8, 9, 9, 8, 8]
Epoch 31: [8, 8, 8, 9, 9, 7, 7, 7, 9, 9, 9, 8, 8, 7, 8, 8, 8, 7, 7, 7]
Epoch 32: [8, 9, 7, 8, 8, 9, 6, 7, 8, 7, 9, 7, 9, 6, 8, 9, 6, 9, 9, 9]
Epoch 33: [8, 8, 9, 5, 7, 7, 5, 6, 9, 6, 5, 9, 9, 6, 5, 5, 6, 9, 5, 5]
Epoch 34: [9, 4, 5, 4, 8, 9, 8, 8, 5, 7, 6, 4, 9, 4, 5, 5, 7, 7, 6, 5]
Epoch 35: [4, 6, 6, 5, 5, 7, 9, 6, 9, 7, 4, 9, 9, 9, 5, 8, 8, 4, 4, 5]
Epoch 36: [5, 5, 3, 7, 6, 5, 8, 9, 9, 5, 9, 7, 8, 9, 7, 9, 6, 8, 4, 3]
Epoch 37: [5, 9, 6, 4, 9, 7, 4, 6, 5, 5, 6, 3, 6, 9, 7, 3, 7, 9, 4, 7]
Epoch 38: [4, 3, 7, 7, 7, 7, 6, 3, 9, 9, 8, 3, 3, 7, 3, 6, 8, 7, 9, 9]
Epoch 39: [6, 5, 9, 5, 6, 4, 4, 3, 4, 3, 6, 8, 9, 7, 4, 7, 6, 6, 4, 4]
Epoch 40: [5, 7, 6, 3, 4, 3, 6, 8, 6, 9, 9, 3, 7, 5, 5, 6, 8, 5, 5, 5]
Epoch 41: [4, 3, 4, 2, 5, 3, 7, 7, 3, 4, 3, 9, 3, 2, 3, 4, 8, 3, 3, 4]
Epoch 42: [8, 8, 2, 5, 6, 9, 7, 7, 2, 2, 9, 9, 4, 3, 3, 6, 9, 6, 6, 6]
Epoch 43: [2, 2, 5, 6, 5, 9, 5, 8, 2, 6, 7, 2, 4, 9, 3, 6, 5, 4, 8, 5]
Epoch 44: [7, 7, 2, 8, 5, 8, 2, 9, 4, 3, 3, 8, 3, 8, 5, 4, 6, 5, 4, 6]
Epoch 45: [2, 5, 6, 7, 4, 9, 6, 5, 4, 7, 9, 7, 5, 9, 3, 8, 9, 6, 5, 2]
Epoch 46: [7, 6, 4, 5, 4, 8, 7, 6, 7, 6, 2, 5, 2, 3, 6, 4, 3, 3, 8, 2]
Epoch 47: [4, 4, 2, 4, 6, 6, 9, 6, 7, 9, 4, 2, 8, 7, 9, 7, 8, 4, 4, 2]
Epoch 48: [7, 8, 4, 5, 6, 4, 2, 4, 5, 9, 6, 2, 2, 6, 8, 9, 5, 4, 7, 7]
Epoch 49: [2, 7, 9, 2, 7, 8, 7, 3, 6, 2, 7, 6, 6, 9, 4, 8, 3, 5, 5, 3]
Epoch 50: [3, 2, 7, 8, 8, 8, 3, 8, 2, 6, 2, 9, 9, 3, 7, 3, 3, 7, 9, 6]
Epoch 51: [2, 4, 3, 3, 4, 4, 3, 4, 8, 3, 3, 5, 3, 6, 6, 4, 2, 9, 7, 3]
Epoch 52: [6, 8, 6, 2, 9, 7, 4, 2, 6, 5, 7, 7, 4, 8, 4, 6, 2, 4, 3, 2]
Epoch 53: [5, 2, 4, 6, 4, 4, 8, 7, 7, 3, 4, 9, 9, 3, 5, 2, 2, 4, 9, 8]
Epoch 54: [9, 6, 4, 3, 2, 7, 3, 4, 2, 7, 7, 2, 8, 2, 9, 9, 5, 2, 7, 3]
Epoch 55: [3, 9, 3, 2, 8, 5, 9, 7, 2, 2, 9, 9, 3, 6, 9, 4, 4, 4, 3, 7]
Epoch 56: [5, 4, 6, 7, 4, 7, 2, 9, 4, 5, 6, 5, 6, 4, 4, 6, 4, 4, 5, 5]
Epoch 57: [7, 5, 9, 3, 7, 8, 9, 6, 2, 4, 6, 9, 8, 8, 5, 4, 2, 8, 8, 2]
Epoch 58: [6, 4, 2, 7, 5, 6, 8, 7, 5, 6, 7, 5, 4, 6, 2, 9, 5, 7, 7, 4]
Epoch 59: [7, 3, 5, 2, 3, 6, 5, 5, 8, 5, 3, 4, 5, 4, 8, 2, 6, 2, 7, 4]
Epoch 60: [4, 2, 7, 3, 8, 3, 7, 7, 8, 4, 9, 2, 5, 2, 3, 2, 4, 8, 4, 7]
Epoch 61: [8, 9, 5, 6, 3, 2, 3, 7, 9, 4, 8, 6, 2, 4, 2, 3, 5, 2, 4, 7]
Epoch 62: [3, 3, 8, 3, 3, 9, 9, 2, 6, 9, 2, 6, 5, 8, 6, 2, 9, 7, 8, 7]
Epoch 63: [4, 2, 7, 9, 9, 3, 9, 3, 8, 3, 4, 8, 5, 6, 4, 2, 2, 5, 8, 3]
Epoch 64: [9, 7, 5, 5, 3, 4, 8, 8, 9, 8, 8, 9, 8, 6, 3, 3, 5, 6, 8, 3]
Epoch 65: [7, 6, 7, 9, 9, 5, 7, 2, 6, 3, 2, 7, 3, 3, 5, 9, 9, 3, 9, 2]
Epoch 66: [7, 5, 4, 9, 3, 5, 3, 2, 5, 9, 2, 6, 3, 5, 4, 7, 5, 8, 9, 2]
Epoch 67: [4, 2, 2, 5, 8, 9, 9, 6, 7, 9, 5, 7, 8, 9, 9, 2, 9, 2, 5, 4]
Epoch 68: [8, 7, 7, 8, 7, 7, 4, 6, 2, 8, 8, 4, 6, 2, 9, 5, 5, 6, 2, 4]
Epoch 69: [4, 6, 5, 2, 6, 9, 4, 4, 3, 9, 5, 6, 7, 7, 4, 8, 4, 8, 8, 8]
Epoch 70: [8, 4, 6, 4, 2, 6, 9, 9, 6, 6, 2, 4, 5, 4, 4, 8, 5, 8, 7, 2]
Epoch 71: [3, 3, 3, 3, 9, 6, 6, 6, 3, 5, 5, 5, 7, 3, 8, 3, 9, 3, 4, 3]
Epoch 72: [4, 9, 3, 2, 7, 8, 5, 4, 6, 9, 8, 4, 6, 2, 5, 7, 2, 3, 8, 7]
Epoch 73: [7, 8, 3, 3, 8, 3, 7, 6, 3, 4, 2, 3, 7, 5, 8, 3, 6, 5, 4, 2]
Epoch 74: [5, 9, 2, 7, 3, 7, 9, 5, 3, 4, 4, 6, 5, 9, 3, 2, 8, 9, 8, 5]
Epoch 75: [2, 8, 4, 9, 5, 3, 2, 9, 5, 7, 3, 6, 3, 7, 5, 7, 5, 2, 6, 6]
Epoch 76: [6, 2, 7, 2, 4, 3, 8, 4, 7, 2, 7, 6, 2, 9, 9, 9, 4, 9, 9, 3]
Epoch 77: [7, 8, 8, 5, 4, 7, 4, 9, 3, 2, 9, 4, 8, 7, 4, 9, 7, 9, 5, 7]
Epoch 78: [9, 9, 6, 9, 9, 7, 7, 3, 4, 8, 3, 5, 8, 9, 5, 5, 5, 8, 8, 2]
Epoch 79: [4, 7, 5, 5, 5, 6, 7, 5, 3, 6, 7, 4, 9, 7, 6, 6, 7, 5, 9, 4]
Epoch 80: [4, 3, 9, 3, 4, 2, 4, 6, 3, 8, 7, 8, 2, 7, 3, 7, 9, 5, 6, 6]
Epoch 81: [7, 9, 4, 3, 7, 5, 3, 7, 9, 7, 5, 7, 6, 3, 2, 9, 5, 7, 5, 8]
Epoch 82: [4, 4, 4, 8, 8, 7, 9, 2, 7, 4, 2, 3, 5, 8, 2, 7, 7, 4, 8, 2]
Epoch 83: [5, 9, 3, 5, 3, 7, 6, 7, 7, 5, 5, 8, 8, 6, 9, 2, 6, 8, 3, 6]
Epoch 84: [4, 3, 5, 2, 7, 3, 4, 7, 9, 8, 7, 7, 7, 6, 3, 2, 9, 3, 7, 6]
Epoch 85: [2, 7, 9, 6, 5, 7, 2, 2, 8, 7, 6, 8, 7, 2, 5, 4, 3, 2, 6, 2]
Epoch 86: [3, 7, 7, 5, 9, 8, 9, 6, 3, 5, 3, 9, 9, 7, 5, 4, 3, 5, 3, 7]
Epoch 87: [8, 8, 7, 5, 4, 9, 2, 2, 6, 7, 5, 4, 4, 2, 6, 6, 2, 9, 4, 6]
Epoch 88: [6, 7, 8, 6, 7, 7, 2, 9, 5, 7, 8, 3, 6, 3, 4, 3, 4, 6, 3, 8]
Epoch 89: [5, 2, 9, 3, 3, 5, 7, 7, 7, 2, 2, 7, 4, 9, 5, 4, 4, 7, 4, 7]
Epoch 90: [7, 7, 7, 6, 3, 2, 8, 6, 4, 8, 8, 6, 9, 9, 9, 3, 9, 6, 6, 5]
Epoch 91: [4, 3, 6, 7, 9, 4, 7, 8, 8, 6, 5, 2, 5, 6, 8, 2, 3, 9, 3, 5]
Epoch 92: [6, 3, 4, 3, 2, 3, 9, 7, 9, 7, 4, 5, 5, 9, 9, 6, 7, 4, 9, 6]
Epoch 93: [2, 9, 9, 8, 4, 6, 8, 9, 7, 8, 7, 9, 4, 4, 8, 6, 3, 6, 2, 8]
Epoch 94: [2, 2, 2, 9, 7, 5, 6, 7, 6, 3, 5, 9, 5, 6, 4, 2, 4, 5, 7, 9]
Epoch 95: [4, 2, 9, 2, 4, 9, 9, 9, 4, 4, 3, 8, 9, 6, 7, 4, 5, 7, 7, 7]
Epoch 96: [9, 6, 9, 5, 6, 9, 6, 4, 9, 4, 7, 7, 7, 4, 9, 9, 2, 7, 9, 4]
Epoch 97: [3, 5, 5, 5, 7, 8, 9, 3, 6, 6, 2, 3, 3, 3, 6, 4, 9, 8, 3, 2]
Epoch 98: [6, 5, 6, 9, 2, 5, 3, 8, 7, 3, 6, 4, 8, 3, 8, 5, 4, 6, 3, 3]
Epoch 99: [5, 3, 6, 6, 7, 9, 6, 5, 7, 6, 3, 5, 3, 9, 3, 2, 9, 2, 6, 5]
Epoch 100: [3, 2, 2, 6, 2, 2, 6, 3, 5, 3, 8, 4, 5, 7, 7, 2, 8, 8, 9, 7]
Epoch 101: [4, 7, 2, 2, 8, 3, 4, 8, 5, 5, 4, 3, 4, 6, 3, 6, 9, 3, 3, 5]
Epoch 102: [2, 8, 4, 9, 3, 4, 9, 7, 2, 8, 2, 4, 3, 8, 2, 8, 9, 8, 7, 4]
Epoch 103: [8, 9, 8, 7, 6, 5, 4, 3, 9, 6, 7, 8, 6, 2, 7, 7, 3, 7, 4, 6]
Epoch 104: [6, 6, 8, 6, 8, 8, 2, 8, 3, 6, 7, 3, 7, 7, 4, 3, 3, 5, 7, 9]
Epoch 105: [3, 6, 9, 4, 8, 2, 7, 8, 3, 4, 6, 9, 7, 7, 6, 7, 8, 3, 5, 2]
Epoch 106: [7, 8, 5, 5, 3, 8, 3, 3, 2, 2, 2, 9, 3, 6, 4, 8, 4, 2, 3, 2]
Epoch 107: [5, 9, 3, 9, 3, 3, 2, 7, 8, 9, 2, 3, 7, 5, 3, 3, 9, 4, 7, 9]
Epoch 108: [2, 8, 7, 9, 4, 9, 2, 2, 3, 9, 2, 3, 9, 6, 8, 9, 5, 3, 3, 5]
Epoch 109: [2, 2, 7, 2, 5, 8, 3, 8, 8, 9, 5, 5, 3, 6, 2, 5, 7, 5, 4, 8]
Epoch 110: [7, 8, 7, 2, 3, 3, 6, 4, 6, 3, 3, 9, 8, 3, 4, 7, 8, 4, 2, 7]
Epoch 111: [5, 3, 7, 3, 8, 8, 8, 7, 2, 4, 7, 3, 4, 9, 9, 4, 9, 7, 2, 7]
Epoch 112: [3, 5, 4, 8, 7, 6, 2, 3, 7, 3, 2, 6, 9, 6, 7, 7, 5, 7, 7, 9]
Epoch 113: [9, 3, 3, 5, 4, 2, 8, 8, 7, 5, 3, 9, 4, 3, 9, 8, 3, 2, 8, 8]
Epoch 114: [8, 4, 7, 9, 8, 3, 3, 3, 2, 5, 5, 3, 2, 6, 4, 9, 4, 7, 2, 7]
Epoch 115: [9, 4, 6, 7, 2, 6, 2, 5, 3, 9, 4, 6, 4, 8, 2, 8, 5, 9, 2, 6]
Epoch 116: [7, 6, 7, 7, 7, 5, 2, 3, 9, 7, 9, 8, 4, 2, 9, 6, 5, 5, 2, 7]
Epoch 117: [9, 7, 5, 9, 6, 4, 7, 7, 3, 3, 3, 5, 5, 9, 4, 6, 3, 3, 3, 4]
Epoch 118: [4, 8, 2, 4, 5, 2, 5, 6, 8, 7, 9, 3, 7, 4, 6, 9, 4, 8, 5, 9]
Epoch 119: [2, 9, 9, 7, 9, 2, 8, 6, 4, 6, 7, 9, 2, 2, 6, 8, 6, 3, 5, 4]

[Alex]

Please provide a reimplementable example so that I can help to locate your problem.

Otherwise, if you don't understand how it works in the details. Going through line by line with pdb or any IDE debugging should help.

[xieydd]

@CanyonWind , Tks, i solve the problem because i make a mistake the logic:
When you use epoch_start_cs, use_all_channels = True, i will close the issue.

[SungAnGweon]

@CanyonWind @xieydd
Hello, I just ran latest code with the script unchanged and found same thing happens.

Traceback (most recent call last):
  File "train_imagenet.py", line 738, in <module>
    main()
  File "train_imagenet.py", line 734, in main
    train(context)
  File "train_imagenet.py", line 710, in train
    err_top1_val, err_top5_val = test(ctx, val_data, epoch)
  File "train_imagenet.py", line 438, in test
    ignore_first_two_cs=opt.ignore_first_two_cs)
  File "/home/nas/Single-Path-One-Shot-NAS-MXNet/oneshot_nas_network.py", line 248, in random_channel_mask
    channel_choice = random.randint(channel_scale_start, len(self.candidate_scales) - 1)
  File "/root/anaconda3/envs/singleshot/lib/python3.6/random.py", line 221, in randint
    return self.randrange(a, b+1)
  File "/root/anaconda3/envs/singleshot/lib/python3.6/random.py", line 199, in randrange
    raise ValueError("empty range for randrange() (%d,%d, %d)" % (istart, istop, width))
ValueError: empty range for randrange() (68,10, -58)

If I use option "--epoch-start-cs 60" and "--cs-warm-up" at the same time, the variable "epoch_after_cs=epoch-opt.epoch_start_cs" becomes negative number in early epochs.
Can you tell me proper usage of these 2 options? Thank you!