A small bug of HKOIterator: IndexError: list index out of range

[Hzzone]

reproduce code:

from nowcasting.hko_iterator import HKOIterator
from nowcasting.config import cfg
valid_hko_iter = HKOIterator(pd_path=cfg.HKO_PD.RAINY_VALID,
                                 sample_mode="sequent",
                                 seq_len=25,
                                 stride=5)
valid_time = 0
while not valid_hko_iter.use_up:
    valid_batch, valid_mask, sample_datetimes, new_start =\
        valid_hko_iter.sample(batch_size=1)
    if valid_batch.shape[1] == 0:
        break
    valid_time += 1

It always reports IndexError: list index out of range error as below:

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-5-4b6c02435465> in <module>
      8 while not valid_hko_iter.use_up:
      9     valid_batch, valid_mask, sample_datetimes, new_start =\
---> 10         valid_hko_iter.sample(batch_size=1)
     11     if valid_batch.shape[1] == 0:
     12         break

~/Downloads/HKO-7/nowcasting/hko_iterator.py in sample(self, batch_size, only_return_datetime)
    445                         datetime_clips.append(datetime_clip)
    446                         break
--> 447         frame_dat, mask_dat = self._load_frames(datetime_clips=datetime_clips)
    448         return frame_dat, mask_dat, datetime_clips, new_start
    449 

~/Downloads/HKO-7/nowcasting/hko_iterator.py in _load_frames(self, datetime_clips)
    323         else:
    324             # Get the first_timestamp and the last_timestamp in the datetime_clips
--> 325             first_timestamp = datetime_clips[-1][-1]
    326             last_timestamp = datetime_clips[0][0]
    327             for i in range(self._seq_len):

IndexError: list index out of range

The error appears because empty datetime_clips list is passed to _load_frames.

So I have added some lines in _load_frames as:

    def _load_frames(self, datetime_clips):
        assert isinstance(datetime_clips, list)
        for clip in datetime_clips:
            assert len(clip) == self._seq_len
        batch_size = len(datetime_clips)
        frame_dat = np.zeros((self._seq_len, batch_size, 1, self._height, self._width),
                                  dtype=np.uint8)
        mask_dat = np.zeros((self._seq_len, batch_size, 1, self._height, self._width),
                                 dtype=np.bool)
        if batch_size==0:
            return frame_dat, mask_dat

The error has disappeared anyway.

[Hzzone]

Hi，学长，我发现使用 HKOEvaluation 进行评估时，balanced_mae or balanced_mse 会一直增加，
原因是 clear_all 方法没有清空前面的结果:

    def clear_all(self):
        self._total_hits[:] = 0
        self._total_misses[:] = 0
        self._total_false_alarms[:] = 0
        self._total_correct_negatives[:] = 0
        self._mse[:] = 0
        self._mae[:] = 0
        self._gdl[:] = 0
        self._ssim[:] = 0
        self._total_batch_num = 0

在整个类中也没有找到清空操作，calculate_stat 计算结果时，每次 clear_all, total_batch_num 也都被清空，只有 bmae 没有。
这样做可能在验证或测试时是对的，但是在训练时一直使用一个 evaulator，会导致 balanced_mae 一直增加。

这可能是个 bug，不过问题不大，很容易修改。