Reduce Memory usage

Question

Reduce Memory usage

stoiver opened this issue 2 years ago · comments

ANUGA is quite a memory hog. For instance when running run_parallel_rectanguar.py (from the examples/parallel directory) with varying --sqrtN values, using a command like::

mpiexec -np 2 time -f "User = %U s System = %S s Maximum RSS = %M kB" python -u run_parallel_rectangular.py --sqrtN 750

we obtained the following results

sqrtN	no of triangles	process 0 RSS memory	User Time
250	250_000	710 MB	6 s
500	1_000_000	2.5 GB	34 s
750	2_250_000	5.1 GB	118 s

We can see very large memory usage for not so large number of triangles. So for a 100_000_000 triangle simulation we would be looking at processor 0 needing 250 GB.

An obvious place to save on memory usage would be to rationalise the memory usage of the Quantity objects. At present we store centroid, edge, vertex and 2 work arrays of size no of triangles. This means 9 x no of triangles. We should be able to get away easily with just centroid, edge and a work array for 5 x no of triangles. A saving of 5/9

Also we store numerous quantities, such as stage, height, elevation, xmom, ymon, xvel, yvel, friction. Probably only need stage, elevation (or height), xmom and ymom. Ie as saving of 4/8.

So total savings could be 5/9 x 4/8 = 20/72 ~ 3/10

To go forward with trying to reduce memory usage, I think it would be necessary to just work with the DE algorithms. An idea would be with say version 4 of anuga we could remove all the old algorithms, and just concentrate on a stream lined version of anuga just using the DE algorithm.

Thoughts?

Petar Milevski · Answer 1 · Sun Dec 18 2022 12:03:08 GMT+0800 (China Standard Time)

Hi Steve that sounds like a great idea. Yes we never use the vertices, so storing them is a waste. I assume elevation (and friction) will be stored once at the start so to be able to calculate depth, unless we are using the scour routines? But a new more efficient anuga version would be a great idea. Regards

…

On Sun, 18 Dec 2022, 14:42 Stephen Roberts, ***@***.***> wrote: @uniomni <https://github.com/uniomni> , @PetarMilevski <https://github.com/PetarMilevski>, @rudyvandrie <https://github.com/rudyvandrie> ANUGA is quite a memory hog. For instance when running run_parallel_rectanguar.py (from the examples/parallel directory) with varying --sqrtN values, using a command like:: mpiexec -np 2 time -f "User = %U s System = %S s Maximum RSS = %M kB" python -u run_parallel_rectangular.py --sqrtN 750 we obtained the following results sqrtN no of triangles process 0 RSS memory User Time 250 250_000 710 MB 6 s 500 1_000_000 2.5 GB 34 s 750 2_250_000 5.1 GB 118 s We can see very large memory usage for not so large number of triangles. So for a 100_000_000 triangle simulation we would be looking at processor 0 needing 250 GB. An obvious place to save on memory usage would be to rationalise the memory usage of the Quantity objects. At present we store centroid, edge, vertex and 2 work arrays of size no of triangles. This means 9 x no of triangles. We should be able to get away easily with just centroid, edge and a work array for 5 x no of triangles. A saving of 5/9 Also we store numerous quantities, such as stage, height, xmom, ymon, xvel, yvel, friction. Probably only need stage, xmom and ymom. Ie as saving of 3/7. So total savings could be 5/9 x 3/7 = 15/63 ~ 1/4 To go forward with trying to reduce memory usage, I think it would be necessary to just work with the DE algorithms. An idea would be with say version 4 of anuga we could remove all the old algorithms, and just concentrate on a stream lined version of anuga just using the DE algorithm. Thoughts? — Reply to this email directly, view it on GitHub <#33>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AETRNNXLJSL3ERUHLZZHMSTWN2B3FANCNFSM6AAAAAATCJLMCI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

rudyvandrie · Answer 2 · Sun Dec 18 2022 20:49:46 GMT+0800 (China Standard Time)

Yes Agreed, Storing Vertices should be optional, for specific analysis where it may be useful. For standard Flood models Centroids should be adequate Rudy

…

On Sun, Dec 18, 2022 at 11:42 AM Stephen Roberts ***@***.***> wrote: @uniomni <https://github.com/uniomni> , @PetarMilevski <https://github.com/PetarMilevski>, @rudyvandrie <https://github.com/rudyvandrie> ANUGA is quite a memory hog. For instance when running run_parallel_rectanguar.py (from the examples/parallel directory) with varying --sqrtN values, using a command like:: mpiexec -np 2 time -f "User = %U s System = %S s Maximum RSS = %M kB" python -u run_parallel_rectangular.py --sqrtN 750 we obtained the following results sqrtN no of triangles process 0 RSS memory User Time 250 250_000 710 MB 6 s 500 1_000_000 2.5 GB 34 s 750 2_250_000 5.1 GB 118 s We can see very large memory usage for not so large number of triangles. So for a 100_000_000 triangle simulation we would be looking at processor 0 needing 250 GB. An obvious place to save on memory usage would be to rationalise the memory usage of the Quantity objects. At present we store centroid, edge, vertex and 2 work arrays of size no of triangles. This means 9 x no of triangles. We should be able to get away easily with just centroid, edge and a work array for 5 x no of triangles. A saving of 5/9 Also we store numerous quantities, such as stage, height, xmom, ymon, xvel, yvel, friction. Probably only need stage, xmom and ymom. Ie as saving of 3/7. So total savings could be 5/9 x 3/7 = 15/63 ~ 1/4 To go forward with trying to reduce memory usage, I think it would be necessary to just work with the DE algorithms. An idea would be with say version 4 of anuga we could remove all the old algorithms, and just concentrate on a stream lined version of anuga just using the DE algorithm. Thoughts? — Reply to this email directly, view it on GitHub <#33>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE26T4LVQM6P6JSEXAWZZCTWN2B3FANCNFSM6AAAAAATCJLMCI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

Samir Shaikh · Answer 3 · Fri Feb 03 2023 19:03:22 GMT+0800 (China Standard Time)

Hi @stoiver ,

In our simulation we are just using DE1 algorithm. Can you give me some hints so as to reduce memory consumption before you make it available in next ANUGA release (4.0.0) ?

Cheers,
Samir

Stephen Roberts · Answer 4 · Sat Feb 04 2023 10:22:11 GMT+0800 (China Standard Time)

Hi @samcom12,

I haven't measured this, but I would think you could halve the memory (on processor 0) by first running a sequential job which creates the sequential domain and dumps the partition, and then run a parallel job which reads in the partition and sets BC etc and then runs the evolve loop.

There is example code in examples/parallel in particular the scripts run_sdpl_rectangular_create_partition_dump.py and run_sdpl_rectangular_load_evolve.py.

As regards reducing memory in the Quantity class, this will be quite a tricky task, so I think you should leave it to us, unless you have a programmer who has some time to work on this. Probably a month of work full time for someone new to ANUGA.

Ole Nielsen · Answer 5 · Sun Feb 05 2023 08:29:27 GMT+0800 (China Standard Time)

I think it makes sense. Only thing that I wonder about is the gradient limiting process. From memory, that's the process we use to assign values to vertices and hence edge values. I also think we use vertex values for visualising models. But we might be able to trade storage for extra computations. But very worthwhile exploring in a new branch. My 5c Ole

…

On Sun, 18 Dec 2022, 14:42 Stephen Roberts, ***@***.***> wrote: @uniomni <https://github.com/uniomni> , @PetarMilevski <https://github.com/PetarMilevski>, @rudyvandrie <https://github.com/rudyvandrie> ANUGA is quite a memory hog. For instance when running run_parallel_rectanguar.py (from the examples/parallel directory) with varying --sqrtN values, using a command like:: mpiexec -np 2 time -f "User = %U s System = %S s Maximum RSS = %M kB" python -u run_parallel_rectangular.py --sqrtN 750 we obtained the following results sqrtN no of triangles process 0 RSS memory User Time 250 250_000 710 MB 6 s 500 1_000_000 2.5 GB 34 s 750 2_250_000 5.1 GB 118 s We can see very large memory usage for not so large number of triangles. So for a 100_000_000 triangle simulation we would be looking at processor 0 needing 250 GB. An obvious place to save on memory usage would be to rationalise the memory usage of the Quantity objects. At present we store centroid, edge, vertex and 2 work arrays of size no of triangles. This means 9 x no of triangles. We should be able to get away easily with just centroid, edge and a work array for 5 x no of triangles. A saving of 5/9 Also we store numerous quantities, such as stage, height, xmom, ymon, xvel, yvel, friction. Probably only need stage, xmom and ymom. Ie as saving of 3/7. So total savings could be 5/9 x 3/7 = 15/63 ~ 1/4 To go forward with trying to reduce memory usage, I think it would be necessary to just work with the DE algorithms. An idea would be with say version 4 of anuga we could remove all the old algorithms, and just concentrate on a stream lined version of anuga just using the DE algorithm. Thoughts? — Reply to this email directly, view it on GitHub <#33>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACJBR2XVAKH7OQ2I6YYFWDWN2B3FANCNFSM6AAAAAATCJLMCI> . You are receiving this because you were mentioned.Message ID: ***@***.***>