Batch inference gives different results

[Suho Lee]

Thanks for your great work.

I've been testing your model for some image examples, but I found that a single image and a batch image(same duplicated image) gives different model output. Here's a code snippet to reproduce the situation:

device = 0
torchscript_file = './PAL4VST/end2end.pt'
img_file = './PAL4VST/0111.jpg'

model = torch.load(torchscript_file).to(device)

# single image prediction
img_tensor = prepare_input(np.array(Image.open(img_file).resize((512, 512))), device) # 1x3x512x512
pal = model(img_tensor).cpu().data.numpy() # 1x1x512x512
print(f"Single image forward : {np.sum(pal)}")

# batch image prediction
img_tensor = [
    prepare_input(np.array(Image.open(img_file).resize((512, 512))), device),
    prepare_input(np.array(Image.open(img_file).resize((512, 512))), device)
]
img_tensor = torch.cat(img_tensor,dim=0) # 2x3x512x512
pals = model(img_tensor).cpu().data.numpy() # 2x1x512x512
print(f"Batch image forward : 1st sample = {np.sum(pals[0])} 2nd sample = {np.sum(pals[1])}")

which gives,

Single image forward : 1881
Batch image forward : 1st sample = 79015 2nd sample = 106755

As far as I know, there is no randomness added to the inference process of this model, so it should always produce the same output for the same image. Could you kindly explain why this is happening?