Was there a performance regression for 3D cases
sbryngelson opened this issue · comments
From @wilfonba there might be a performance regression for 2D vs. 3D cases at some point. Bisection of tags/old commits to see if this occurred and if so, what caused it, is required.
I ran some tests on Phoenix using the 3D_sphbubblecollapse example case and got the following average time/step for 8m cells (200x200x200 grid) on 4 V100 GPUs:
MFC-4.0.0 - 0.2599 s/step
MFC-4.6.0 - 0.2612 s/step
for a 0.5% difference. Running 2D_advection with 8m (2000x4000 grid) cells on 4 V100 GPUs gives the following results:
MFC-4.0.0 - 0.1012 s/step
MFC-4.6.0 - 0.1063 s/step
which is a 5% increase, which is a bit more concerning. Lastly, I ran 2D_advection with 2m cells (1000x2000 grid) on 1 V100 GPU to mitigate any communication time and got the following results:
MFC-4.0.0 - 3.9197e-2 s/step
MFC-4.6.0 - 4.1292e-2 s/step
which is also a 5% increase in run time. In conclusion, there's a chance there has been a slight slowdown since MFC-4.0.0, but the communication times (I'm guessing) have also skyrocketed uniformly in 2 and 3 dimensions with multiple ranks. Single-rank simulations perform as expected.
This is very useful. Your 5% is hard to gauge the importance of since the seconds per step are so short. Thus, from these quantities, I would not raise any concerns. It seems like we were just experiencing the performance one should expect when moving from 2D -> 3D or introducing different boundary conditions, we just didn't actually expect it. Closing this issue for now. Thanks @wilfonba .