Same seed yields different results on GPU
Quasimondo opened this issue · comments
Running texturizer with the same seed results in slightly different results when repeated. I would expect the same seed to generated the identical texture every time. Here are three examples:
Looks like this is caused by some non-deterministic CUDA functions - since when running the same seed on the CPU the results are identical.
From https://pytorch.org/docs/stable/notes/randomness.html:
There are some PyTorch functions that use CUDA functions that can be a source of nondeterminism. One class of such CUDA functions are atomic operations, in particular atomicAdd, which can lead to the order of additions being nondetermnistic. Because floating-point addition is not perfectly associative for floating-point operands, atomicAdd with floating-point operands can introduce different floating-point rounding errors on each evaluation, which introduces a source of nondeterministic variance (aka noise) in the result.
Try torch.backends.cudnn.deterministic = True ?
I use this all the time, appears to save memory.
Ah, sorry I didn't mention that of course I had tried all the usual settings mentioned in the docs. It does not help.
Oh yes... good to know.
It's good to know the CPU results are identical, that rules out issues & bugs directly in this library ;-)
It's not clear to me what is causing the non-determinism on GPU. torch.nn.Conv2d should become deterministic with torch.backends.cudnn.deterministic = True, but the other candidate in the encoder is only torch.nn.AvgPool2d — which has no warnings in the documentation about determinism.
It may be that the L-BFGS optimizer itself also introduces some non-determinism. Testing a plain SGD optimizer would rule that out...
I had first suspected the LBFGS solver and tentatively replaced it with Adam. The results look like crap, but they also differ when run on CUDA with the same seed.
I forgot to quote the most important line from that "Reprocibility" link above:
**
"There is currently no simple way of avoiding nondeterminism in these functions."
**
It's a pity since I was trying to do some seed interpolation animations as well as a sliding window test for getting larger textures than fit in the GPU, but both of them do not work with CUDA since they change on every iteration.
The interpolation would not be deterministic anyway, as soon as you change the content of the starting tensor, it would introduce different numbers and hence instability.
https://arxiv.org/abs/1705.02092v1
As for larger textures, I have a variety of code for tiling but not with gram matrices yet... I will see what's possible!