Python kernel stops running and freezes when repeatedly running backward passes

[abhinavvs]

I am trying to use Redner to perform object localization - a simple single parameter optimization. I am able to successfully use redner to compute the forward pass. However, the python kernel freezes after some iterations of optimization. The exact number of iterations varies (usually, anywhere between 50-150 iterations), but it happens every time.

There are no errors thrown by Jupyter on the terminal console, but the program just stops running (or generating any intermediate outputs). I'd have to restart the kernel every time this happens. I am including a simple working example here: example.zip. Please try running MLE_Gaussian_redner_1D_estimation.ipynb.

I am using redner-gpu on NVIDIA Titan-V. I checked the GPU memory usage using nvidia-smi command while the script was running and I saw that there was still a lot of free memory.

Thanks in advance for your help!

---

Update: I tried varying the samples per pixel used for rendering and also tried clearing the cache every few iterations using torch.cuda.empty_cache(). Neither of these seems to help!

[Dusk]

Was having the same problem on one of my scripts updating my gpu driver to NVIDIA-Linux-x86_64-510.47.03 fixed the problem.