(potential) memory leak in GAP calculator (ASE) used with quippy

[Richard Jana]

I have a database of structures stored in extended xyz format, appended in a single file and would like to calculate their GAP energies, forces etc. for all of them using ASE via quippy. Doing this, I run out of memory at some point, as was described in QUIP issue 235. Upgrading f90wrap to the newest version (opposed to the one I got from pip before) led me to the error discussed in issue 137. So I followed the proposed fix there and downgraded my f90wrap version to a580827, which should address the problem. However, then I got the memory leak again...
I checked that I have no other f90wrap versions on my machine, and the fact that I can reproduce the error from issue 137 also indicates that I am using the correct version of f90wrap. Just to be sure, I also checked that the changes in source code were there.
Therefore, it seems to me like there might be another problem in the code, for the way I intended to use it.
Here is a minimal ASE script for my problem:
`
import gc
import numpy as np
from ase import Atoms
from ase.io import read, write
import quippy

    gap_calculator = quippy.potential.Potential('IP GAP', param_filename='carbon.xml')
    
    energy_GAP = []
    
    for i in range(1000): # read the first 1000 configurations; this is more than enough to fill my memory
        try:
            atoms = read('DB.xyz', format = 'extxyz', index = i)
        except:
            break
    
        n_atoms = atoms.get_global_number_of_atoms()
        atoms.calc = gap_calculator
        energy_GAP = np.append(energy_GAP, atoms.get_total_energy()/n_atoms)
    
        del atoms
        gc.collect()

`
Note that my potential contains TurboGAP descriptors. I don't know if this is relevant, but now you know.

[bernstei]

FWIW, I just checked with one of my (2b + 2xSOAP, not TurboSOAP) GAPs, and memory use was constant. It's either a turbo-SOAP issue, or there's something else wrong. I believe I'm using the latest QUIP and GAP, and f90wrap --version returns 0.2.2.

Also, note that unless you have some other reason to write the code the way you did, it's possible to be a lot more compact, e.g.:

for at in iread('DB.xyz', '0:1000'):
    at.calc = gap_calculator
    energy_GAP.append(at.get_total_energy()/len(at))

If nothing else, that's definitely going to be a lot more efficient with the I/O (since it won't re-open the file and seek to the correct config 1000 times).

[James Kermode]

Would you be willing to try turning off TurboSOAP to see if the leak goes away?

[Richard Jana]

Of course! I retrained my potential, keeping everything the same, except that I removed the soap_turbo descriptors. (So now I have only 2b and 3b.) Now I don't seem to have the issue anymore (or rather, the issue only seems to come up when I use soap_turbo descriptors).

@bernstei: Thanks for the tip on reading my input. I will definitely use that!

[James Kermode]

Thanks, that’s good to know. It could be that without the multi body term the leak is just much slower, so could you try one final test: switch turbo soap for a standard SOAP descriptor?

[Richard Jana]

After recompiling the f90wrap version and quippy/QUIP again, now all potentials (with and without soap_turbo) suddenly seem to work. I am really not sure why, as I thought I had tried exactly this setup before (but apparently this was not the case).

Sorry for causing all this trouble and a big thank you to all who helped!

Edit: I wasn't sure if I was supposed to close this issue just now; I reopened it to enable any possible answers. Let me know when can close it.

[James Kermode]

No worries, glad this is resolved. I'll close the issue, but do reopen if the memory leak comes back.