Codeveloping MFEM with PETSc

[Alex Tyler Chapman]

Since Serac has some logic for finding petsc in it's SetupSeracThirdParty.cmake, it creates some problems when codeveloping Serac in LiDO. It has to do with MFEM's FindPETSc.cmake module. I need to look into this more once I start working on building PETSc for Serac.

    if(PETSC_DIR)
        serac_assert_is_directory(VARIABLE_NAME PETSC_DIR)
        include(${CMAKE_CURRENT_LIST_DIR}/FindPETSc.cmake)
        message(STATUS "PETSc support is ON")
        set(PETSC_FOUND TRUE)
    else()
        message(STATUS "PETSc support is OFF")
        set(PETSC_FOUND FALSE)
    endif()

CMake Error at smith/serac/mfem/config/cmake/modules/FindPETSc.cmake:147 (message):
  The pair
  PETSC_DIR=/usr/WS1/topopt/libs/lido/toss_4_x86_64_ib/2024_02_16_16_37_47/linux-rhel8-ivybridge/clang-14.0.6/petsc-3.20.4-iwtouyzn3b2zdjpofsc6n637ae52lsbr
  PETSC_ARCH=arch-linux2-c-debug do not specify a valid PETSc installation
Call Stack (most recent call first):
  smith/serac/mfem/CMakeLists.txt:254 (find_package)


CMake Error at /usr/tce/backend/installations/linux-rhel8-x86_64/gcc-10.3.1/cmake-3.23.1-mdfqd2l7c33zg7xcvqizwz25vqmp7jfw/share/cmake-3.23/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  PETSc could not be found.  Be sure to set PETSC_DIR and PETSC_ARCH.
  (missing: PETSC_INCLUDES PETSC_EXECUTABLE_RUNS)
Call Stack (most recent call first):
  /usr/tce/backend/installations/linux-rhel8-x86_64/gcc-10.3.1/cmake-3.23.1-mdfqd2l7c33zg7xcvqizwz25vqmp7jfw/share/cmake-3.23/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  smith/serac/mfem/config/cmake/modules/FindPETSc.cmake:349 (find_package_handle_standard_args)
  smith/serac/mfem/CMakeLists.txt:254 (find_package)

[Alex Tyler Chapman]

Update: the issue is MFEM's FindPETSc.cmake module is extremely out of date. It was last updated 8 years ago. It's possible an older version of petsc (@3.5) will resolve the issue. For now I'll just disable MFEM_USE_PETSC for codevelop builds.

[Alex Tyler Chapman]

Turns out Serac's FindPETSc.cmake was conflicting with MFEM's, and only MFEM's should be called during codevelop. Also, there is a way to bypass their build-checks by setting PETSC_EXECUTABLE_RUNS.

[Alex Tyler Chapman]

For some reason, slepc complains about undefined petsc and arpack symbols. In Serac's cmake module for slepc, I put in some logic to add the needed libs and include dirs. However, MFEM's cmake module does not do this. Fixing this is a challenge, since MFEM does not seem to create a petsc or slepc target that I am aware of. This makes it hard to add these libs and include dirs i need to add to slepc.

Closing issue for now in favor of simply not configuring MFEM with petsc/ slepc. It will still be possible to use petsc/ slepc separate from MFEM.

This may be revisited in the future, so I'll add this block that's needed in the MFEM codevelop session to even begin to configure MFEM with petsc/ slepc in a codevelop build.

if(PETSC_DIR)
    set(MFEM_USE_PETSC ON CACHE BOOL "")
    set(PETSC_ARCH "" CACHE STRING "")
    set(PETSC_EXECUTABLE_RUNS "ON" CACHE BOOL "") # Skips build checks
    if(SLEPC_DIR)
        set(MFEM_USE_SLEPC ON CACHE BOOL "")
        set(SLEPC_ARCH "" CACHE STRING "")
        set(SLEPC_VERSION_OK "TRUE" CACHE BOOL "") # Skips build checks
    endif()
else()
    set(MFEM_USE_PETSC OFF CACHE BOOL "")
    set(MFEM_USE_SLEPC OFF CACHE BOOL "")
endif()

...

# add mfem subdirectory