Semantic segmentation output rgb values do not match input class values

[Nir]

I followed "Semantic Segmentation of 150 classes of objects With 5 Lines of Code" here.
Works well but the output labeled pixels 'blend' in boundaries between patches, so in the output there are more unique values than just the original class rgb values of ade20k.
I don't know if it's meant to be this way, and wonder if there is a way to get only the input class values in the output.
My inputs are 640x640 Google Street View jpg images.