A problem with the code running on the PGM dataset
wangm-buaa opened this issue · comments
The result of using this code on the RAVEN dataset is roughly the same as the data shown in the paper, but the performance of this code on the PGM data set is much different from the paper (we only got about 30% accuracy)
We noticed that the image data(16x160x160) of the RAVEN dataset is different from the image data (160x160x16) of the PGM dataset. Is there any other details to be considered when running on the PGM dataset?
Code provided：
data = np.load(data_path)
image = data["image"].reshape(16, 160, 160)
target = data["target"]
if self.img_size != 160:
resize_image = []
for idx in range(16):
resize_image.append(misc.imresize(image[idx, :, :], (self.img_size, self.img_size)))
image = np.stack(resize_image)
image = torch.tensor(image, dtype=torch.float)
target = torch.tensor(target, dtype=torch.long)
After modification：
data = np.load(data_path)
image = data["image"].reshape(160, 160,16)
target = data["target"]
if self.img_size != 160:
resize_image = []
for idx in range(16):
resize_image.append(misc.imresize(image[:, :, idx], (self.img_size, self.img_size)))
image = np.stack(resize_image)
image = torch.tensor(image, dtype=torch.float)
target = torch.tensor(target, dtype=torch.long)
We noticed the performance is reported for the neutral split, and the code has already run 200 epochs on this part. How many epochs may be needed to achieve the effect of the display?
We did not run 200 epochs to get the performance. Usually after 30 epochs, it plateaus. We ran the code on a 4-GPU server. I'm not sure if batch norm would affect it, as we did not use synchronized batch norm. Em, other factors to consider, maybe try tuning the learning rate.
If your team uses parameters that are different from the default values in the code, could you please tell us your settings (such as batch-size and learning rate)?
We also have a 4-GPU server.With great interest to your research, we are looking forward to a local reproduction on PGM dataset and promotion. Thank you very much for your patience and enthusiasm.
We tried the different learning rates you provided. Each learning rate was trained with 200 epochs on the neutral part of PGM, and the best result was 51.19%(lr=1e-3). We plan to publish the paper according to this data if you have no other suggestions here.