Memory leak on GPU chi2Profile()

[Marco Tazzari]

I am using galario for a real case usage on high resolution data: it works pretty nicely in terms of speed, but I think there is a memory leak :….
I am running an mcmc fit using emcee’s MPI pool with 12 processes on my desktop.
At the beginning the base memory usage by each of the 12 processes on the GPU is ~290MB, peaking to ~490MB during operation.
But as time passes the base usage increases from 290MB to >380MB, and in the end it goes out of memory.
Every ~10 chi2Profile evaluations it increases by ~2MB.
I am using matrices 4096k x 4096k.

From a quick look I couldn’t find missing cudaFree.

[Marco Tazzari]

I think the issue could be the definition of the cublas_handle() here:

galario/src/galario.cpp

Line 140 in 4f59e57

 static bool initialized = false; 

I notice that initialized is never set to true. Is this correct behaviour? To me it seems could be the cause of the leak: if initialized is never set to true, then at each call of cublas_handle() a new handle will be created and assigned to the same address, which makes it impossible to free the previous one.
Moreover, cublas_handle() is called by chi2Profile() through a call to reduce_chi2_d():

galario/src/galario.cpp

Line 1378 in 4f59e57

 CUBLASNRM2(cublas_handle(), nd, fint, 1, chi2);