Tensor not dropped after another tensor was assigned to variable in cuda
daguix opened this issue · comments
Hello,
When I execute this code
`let tensor1 = Tensor::randn(0f32, 1f32, &[1024,1024,1024], &gpu)?;
let tensor1 = candle_nn::ops::softmax_last_dim(&tensor1)?;
let tensor2 = Tensor::randn(0f32, 1f32, &[1024,1024,1024], &gpu)?;`
It takes 12GB of VRAM instead of 8GB. It runs into OOM error in cuda. The old tensor1 is not dropped in second line.
Meanwhile with this code:
`let tensor1 = Tensor::randn(0f32, 1f32, &[1024,1024,1024], &gpu)?;
let tensor2 = candle_nn::ops::softmax_last_dim(&tensor1)?;
drop(tensor1);
let tensor3 = Tensor::randn(0f32, 1f32, &[1024,1024,1024], &gpu)?;`
the memory is correctly cleaned.
Same with cpu btw
That's not really specific to candle but rather a rust behavior: variables only get dropped at the end of the scope (it also applies to mutex guards and all other RAII use cases).