Bug: GIT_SUBMODULES_RECURSE and GIT_SUBMODULES are ignored

Question

Bug: GIT_SUBMODULES_RECURSE and GIT_SUBMODULES are ignored

ethindp opened this issue a year ago · comments

I have a CMake project that has private submodules. The project's CMake configuration process will acquire the public submodules that I need. Therefore, I've used the following git-related options:

    GIT_SUBMODULES ""
    GIT_SUBMODULES_RECURSE FALSE

According to the CMake documentation, with this set, FetchContent_Declare (and, by extension, ExternalProject_Add) will not clone submodules, as long as CMP0096 is set to NEW. So I did that at the top of my project:

cmake_policy(VERSION 3.25)
cmake_policy(SET CMP0096 NEW)

However, the arguments I set in CPMAddPackage (GIT_SUBMODULES and GIT_SUBMODULES_RECURSE) are ignored, and all submodules are cloned anyway. Is this a bug in CPM? Logically this should work perfectly fine, but it isn't for some reason.

Ethin Probst · Answer 1 · Sat Apr 15 2023 03:33:32 GMT+0800 (China Standard Time)

Okay, so it looks like CPMAddPackage doesn't actually take all possible content options. CPM should probably be updated to take into account all options that ExternalProject/FetchContent accepts.

Lars Melchior · Answer 2 · Wed Apr 19 2023 00:05:50 GMT+0800 (China Standard Time)

CPM should already forward all additional arguments to FetchContent_Declare, so hard to say what's not working here. Can you confirm that CMake behaves as intended using the same arguments passed directly to FetchContent?

Ethin Probst · Answer 3 · Wed Apr 19 2023 01:00:17 GMT+0800 (China Standard Time)

@TheLartians Yes, can confirm. When I replace the CPMAddPackage line with FetchContent_Declare, it works fine. I do have to set CMP0096 to NEW, but other than that I get the expected behavior.

The reason, as I discovered, is that CPMAddPackage does not in fact forward all arguments to FetchContent_Declare. It only extracts the NAME, FORCE, VERSION, GIT_TAG, DOWNLOAD_ONLY, GITHUB_REPOSITORY, GITLAB_REPOSITORY, BITBUCKET_REPOSITORY, GIT_REPOSITORY, SOURCE_DIR, DOWNLOAD_COMMAND, FIND_PACKAGE_ARGUMENTS, NO_CACHE, SYSTEM, GIT_SHALLOW, EXCLUDE_FROM_ALL, SOURCE_SUBDIR, URL, and OPTIONS arguments, but FetchContent_Declare(and by extension,ExternalProject_Add) supports many more arguments than these. At least, that's my understanding of the code. I'm not positive how cmake_parse_arguments works, so for all I know I could be wrong.

Lars Melchior · Answer 4 · Wed Apr 19 2023 02:30:12 GMT+0800 (China Standard Time)

In fact from my understanding it should capture all arguments. From the CMake docs:

All remaining arguments are collected in a variable _UNPARSED_ARGUMENTS that will be undefined if all arguments were recognized.

Which we should later forward to FetchContent. So I'm unsure why it isn't working in this case. Can you share a minimal example to reproduce?

Ethin Probst · Answer 5 · Wed Apr 19 2023 03:13:49 GMT+0800 (China Standard Time)

@TheLartians Sure thing. I'm trying to use a project that has some (private) submodules. So I can't just recursively fetch everything; I don't have rights to all of the submodules. The CMake project, therefore, fetches the modules that I actually need, and excludes those that are private, unless I explicitly opt into it doing so. So, I tried this using CPM:

CPMAddPackage(
    NAME nabla
    VERSION 0.4-beta
    GIT_REPOSITORY https://github.com/devsh-graphics-programming/nabla
    GIT_TAG v0.4-beta
    GIT_SUBMODULES ""
    GIT_SUBMODULES_RECURSE FALSE
    GIT_PROGRESS TRUE
    OPTIONS
    "NBL_COMPILE_WITH_CUDA ON"
    "NBL_BUILD_DPL ON"
    "NBL_PCH ON"
    "NBL_FAST_MATH ON"
    "NBL_BUILD_EXAMPLES OFF"
    "NBL_BUILD_TOOLS OFF"
    "NBL_BUILD_MITSUBA_LOADER ON"
    "NBL_BUILD_OPTIX ON"
    "NBL_EMBED_BUILTIN_RESOURCES ON"
)

Doing that still caused Git to fetch all submodules, even those that were private. Which is not my desired behavior. A minimal example for a project might be something like:

cmake_minimum_required(VERSION 3.25)
cmake_policy(VERSION 3.25)
cmake_policy(SET CMP0096 NEW)
cmake_policy(SET CMP0077 NEW)
project(game VERSION 0.0.1 LANGUAGES CXX)
include(CheckPIESupported)
check_pie_supported()
if(NOT CMAKE_MAXIMUM_RECURSION_DEPTH)
  set(CMAKE_MAXIMUM_RECURSION_DEPTH 2000)
endif()
add_executable(game
    main.cpp
)
# Fetch external targets
set(CPM_DOWNLOAD_VERSION 0.38.1)

if(CPM_SOURCE_CACHE)
    set(CPM_DOWNLOAD_LOCATION "${CPM_SOURCE_CACHE}/cpm/CPM_${CPM_DOWNLOAD_VERSION}.cmake")
elseif(DEFINED ENV{CPM_SOURCE_CACHE})
    set(CPM_DOWNLOAD_LOCATION "$ENV{CPM_SOURCE_CACHE}/cpm/CPM_${CPM_DOWNLOAD_VERSION}.cmake")
else()
    set(CPM_DOWNLOAD_LOCATION "${CMAKE_BINARY_DIR}/cmake/CPM_${CPM_DOWNLOAD_VERSION}.cmake")
endif()

if(NOT (EXISTS ${CPM_DOWNLOAD_LOCATION}))
    message(STATUS "Downloading CPM.cmake to ${CPM_DOWNLOAD_LOCATION}")
    file(DOWNLOAD
        https://github.com/cpm-cmake/CPM.cmake/releases/download/v${CPM_DOWNLOAD_VERSION}/CPM.cmake
        ${CPM_DOWNLOAD_LOCATION}
    )
endif()

include(${CPM_DOWNLOAD_LOCATION})

CPMAddPackage(
    NAME nabla
    VERSION 0.4-beta
    GIT_REPOSITORY https://github.com/devsh-graphics-programming/nabla
    GIT_TAG v0.4-beta
    GIT_SUBMODULES ""
    GIT_SUBMODULES_RECURSE FALSE
    GIT_PROGRESS TRUE
    OPTIONS
    "NBL_COMPILE_WITH_CUDA ON"
    "NBL_BUILD_DPL ON"
    "NBL_PCH ON"
    "NBL_FAST_MATH ON"
    "NBL_BUILD_EXAMPLES OFF"
    "NBL_BUILD_TOOLS OFF"
    "NBL_BUILD_MITSUBA_LOADER ON"
    "NBL_BUILD_OPTIX ON"
    "NBL_EMBED_BUILTIN_RESOURCES ON"
)
include(${CMAKE_CURRENT_BINARY_DIR}/_deps/nabla-src/cmake/build/AddNablaModule.cmake)
addNablaModule(game "${CMAKE_CURRENT_BINARY_DIR}/_deps/nabla-build/install")
target_compile_features(game PUBLIC cxx_std_20)
set_target_properties(game PROPERTIES
    CXX_EXTENSIONS OFF
    POSITION_INDEPENDENT_CODE TRUE
    C_STANDARD 18
    C_STANDARD_REQUIRED 18
    CXX_SCAN_FOR_MODULES ON
    CXX_STANDARD 20
    CXX_STANDARD_REQUIRED 20
    INTERPROCEDURAL_OPTIMIZATION ON
   PCH_INSTANTIATE_TEMPLATES ON
    PCH_WARN_INVALID ON
    UNITY_BUILD ON
    UNITY_BUILD_UNIQUE_ID "GAME"
)

(I might've made some errors, I hope not.) The source code in main.cpp can be anything, since CMake fails the configure stage.

Ethin Probst · Answer 6 · Wed Apr 19 2023 03:16:13 GMT+0800 (China Standard Time)

@TheLartians I should also note that you'll need the Cuda toolkit and NVIDIA OptiX for this to work. You can turn that off if you don't have them or you don't want to go get them just for this. The configure stage will take quite a while anyway, though.

Lars Melchior · Answer 7 · Wed Apr 19 2023 03:23:00 GMT+0800 (China Standard Time)

Super, thanks for the reproducer!
Now that I see it I believe this is related an issue currently being worked on where empty string parameters (eg the "" in GIT_SUBMODULES "") get lost in the CMake parsing toolchain.

Hopefully the fix will be released soon, but in the meanwhile could you try replacing the contents of CPM.cmake with this development version and see if that solves your issue?

Craig Hutchinson · Answer 8 · Tue Sep 26 2023 19:52:53 GMT+0800 (China Standard Time)

@ethindp I feel this bug was prematurely closed, the Fix PR hasn't made progress so this bug should be open so its visible to others.