Simulation: Reduce memory footprint for parallel execution
prisae opened this issue · comments
Currently, emg3d simply uses multiprocessing to run jobs in parallel. This might be enough, not sure yet. However, currently it also copies over the simulation to each job. This very soon results in memory issues when many jobs are run in parallel. Also, there seems to be a memory leak on some platforms, which we are not sure yet what the reason is.
Tasks:
- Only send particular model and source to the processes, not entire simulations.
IMPORTANT: Make sure that an initial electric field can be provided. This can have a huge speed-up for inversions. - See if this improves the memory leakage
- Implement flag for file-driven execution (fields are saved to and read from disk) - if hundreds of src-freq pairs are to be computed this will be the only viable solution.
Simulation is not shared any longer. File-driven flag might still be of interest.