How to configure a composite solve with hypre's linear solvers?
zongy17 opened this issue · comments
Hi, I am new to Castro, and don't know how to use hypre for composite solve. By "composite", I mean that the unknowns on all AMR levels are solved simultaneously.
When I look into the executable cases, I found these comments in Castro/Exec/radiation_tests/RadSourceTest/inputs-cooling:
# solver flag values < 100 use HypreABec, support symmetric matrices only
# solver flag values >= 100 use HypreMultiABec, support nonsymmetric matrices
# 0 SMG
# 1 PFMG
# 100 AMG
# 102 GMRES using ParCSR interface
# 103 GMRES using SStruct interface
# 104 GMRES using AMG as preconditioner
radsolve.level_solver_flag = 0 # can be any supported hypre solver flag
# if use_hypre_multilevel == 0, use CompSolver with an internal hypre
# level solver determined by multilevel_solver_flag.
# if use_hypre_multilevel == 1, use HypreMultiABec directly,
# multilevel_solver_flag must then be >= 100.
radsolve.use_hypre_multilevel = 0
radsolve.multilevel_solver_flag = 0
But no matter how I change these three parameters, the solvers are always on a single level. I verified it by printing the crse_level, fine_level and nparts in the constructor function of HypreMultiABec (located at Line 613 of Castro/Source/radiation/HypreMultiABec.cpp). It showed that crse_level==fine_level and nparts==1 all the time. I suppose that in a composite solve, nparts should be the number of AMR levels. So I do not understand what configuration should be set in input files to do a composite solve. Could somebody help me to figure it out? Thanks!
Hypre is only used in Castro for the implicit part of the radiation solve and that is only done a level at a time.
Thanks for your comments! I tried reading the Castro's paper (https://arxiv.org/abs/1105.2466), and noticed that in Section 4, it said
The AMR version of the implicit radiation diffusion update is based on Howell & Greenough (2003), but with
several important differences: The present algorithm is fully-implicit, not time-centered. The optional multilevel
linear solve at the beginning of each coarse time step is no longer included—this feature was introduced to improve accuracy but we now consider it unnecessary in most cases. Finally, the multilevel linear solve for flux synchronization between coarse and fine levels is replaced by a new algorithm we call the “deferred sync.” These changes entirely eliminate the need to compute linear system solutions coupling different levels of the AMR hierarchy, while not compromising conservation of total
energy. Performance is significantly improved because multilevel linear solvers tend to be more complex and
expensive than those for single-level systems.
So, Castro does NOT do a multi-AMR level composite solve (i.e., all unknowns on different AMR levels are solved simultaneously) at any time, because consecutive single-level solve is more efficient. The single-level solve means that an external multigrid solver could be used for each AMR level independently. For Structured AMR problems, single-level solve one-by-one is enough.
Is my understanding correct?
For gravity, the multigrid does composite solves.
The radiation stuff is not widely used, and I don't fully remember the details.
closing due to lack of activity