how to using local source to build instead of codes by GIT_REPOSITORY

Question

how to using local source to build instead of codes by GIT_REPOSITORY

9woods123 opened this issue a year ago · comments

Hello, great work.
I am using this repository when building a ros package.
I noticed that the source is download by ExternalProject_Add() from https://github.com/grpc/grpc.git.

ExternalProject_Add(
  build_grpc
  PREFIX grpc
  GIT_REPOSITORY "https://github.com/grpc/grpc.git"
  GIT_TAG v1.34.1
  GIT_CONFIG submodule.recurse=1 submodule.fetchJobs=10
  GIT_SHALLOW 1
  SOURCE_DIR grpc_build
  CONFIGURE_COMMAND ""
  BUILD_IN_SOURCE 1
  BUILD_COMMAND $(MAKE) CONFIG=opt EMBED_ZLIB=true
  INSTALL_COMMAND ""
)

While the process is clear, we've encountered issues with the downloads consistently failing, causing interruptions in the package building process. This is especially problematic in environments with poor or no network connectivity.

Considering this, if we have a locally downloaded source available, is it possible to modify the CMakeLists.txt to use the local source for the build?

I adjust the CMakeLists.txt like this, to keep a same file structure, but interrupt with error.

ExternalProject_Add(
  build_grpc
  PREFIX grpc
  SOURCE_DIR /source_path
  CONFIGURE_COMMAND ""
  BUILD_IN_SOURCE 1
  BUILD_COMMAND $(MAKE) CONFIG=opt EMBED_ZLIB=true
  INSTALL_COMMAND ""
)

set(GRPC_BUILD_DIR ${CMAKE_CURRENT_BINARY_DIR}/grpc_build)
file(MAKE_DIRECTORY ${GRPC_BUILD_DIR})


    execute_process(
      COMMAND rsync -av /source_path/ ${GRPC_BUILD_DIR}
      RESULT_VARIABLE RSYNC_RESULT
    )
    if (NOT RSYNC_RESULT EQUAL "0")
      message(FATAL_ERROR "Failed to copy external source code.")
    endif()

error:

DEPENDENCY ERROR

You are missing system dependencies that are essential to build grpc,
and the third_party directory doesn't have them:

  cares openssl

Installing the development packages for your system will solve
this issue. Please consult INSTALL to get more information.

If you need information about why these tests failed, run:

  make run_dep_checks

I appreciate your assistance.

Shengye Wang · Answer 1 · Wed Aug 30 2023 17:12:14 GMT+0800 (China Standard Time)

Thanks for reaching out! I can see the difference between the two ExternalProject_Adds is GIT_CONFIG submodule.recurse=1 submodule.fetchJobs=10.

You might want to run git submodule update --init in your local directory (/source_path).

Tianyou Yu · Answer 2 · Wed Aug 30 2023 17:34:11 GMT+0800 (China Standard Time)

Thanks for your kind reply.

The process in your original repository is download the grpc after exectue catkin build and then make it.
This method is clear and convenient, except when the network is not good.

I guess that if we have a previously downloaded source files from https://github.com/grpc/grpc.git and use it to build the catkin_grpc instead of git clone when ExternalProject_Add. how to adjust the CMakeList.txt of catkin_grpc ?

The idea is that robotic platform is not always in a good network, but we need to build the ros package. The method of using the ExternalProject_Add and GIT_REPOSITORY become invalid.

Shengye Wang · Answer 3 · Wed Aug 30 2023 17:38:27 GMT+0800 (China Standard Time)

I think your original proposal

ExternalProject_Add(
  build_grpc
  PREFIX grpc
  SOURCE_DIR /source_path
  CONFIGURE_COMMAND ""
  BUILD_IN_SOURCE 1
  BUILD_COMMAND $(MAKE) CONFIG=opt EMBED_ZLIB=true
  INSTALL_COMMAND ""
)

makes sense. It's just in addition to git clone https://github.com/grpc/grpc.git /your/local/source_path, you also need cd /your/local/source_path; git submodule update --init

Tianyou Yu · Answer 4 · Wed Aug 30 2023 17:46:16 GMT+0800 (China Standard Time)

yeah, the original proposal does make sense, but I noticed that SOURCE_DIR grpc_build make a dictionary named grpc_build which is then used in the subsequent actions.

add_custom_command(
  TARGET build_grpc
  POST_BUILD
  COMMAND ${CMAKE_COMMAND} -E make_directory ${GRPC_INCLUDE_DESTINATION}
  COMMAND ${CMAKE_COMMAND} -E make_directory ${GRPC_LIB_DESTINATION}
  COMMAND ${CMAKE_COMMAND} -E make_directory ${GRPC_BIN_DESTINATION}
  COMMAND
    ${RSYNC}
      --prune-empty-dirs --archive
      ${CMAKE_CURRENT_BINARY_DIR}/grpc_build/libs/opt/
      ${GRPC_LIB_DESTINATION}/
  COMMAND
    ${RSYNC}
      --prune-empty-dirs --archive
      ${CMAKE_CURRENT_BINARY_DIR}/grpc_build/bins/opt/
      ${GRPC_BIN_DESTINATION}/
  COMMAND
    ${CMAKE_COMMAND} -E make_directory ${GRPC_INCLUDE_DESTINATION}/src/proto
  COMMAND
    ${RSYNC}
      --prune-empty-dirs --archive
      --include="*/" --include="*.proto" --exclude="*"
      ${CMAKE_CURRENT_BINARY_DIR}/grpc_build/src/proto/
      ${GRPC_INCLUDE_DESTINATION}/src/proto/
  COMMAND
    ${RSYNC}
      --prune-empty-dirs --archive
      --include="*/" --include="*.h" --exclude="*"
      ${CMAKE_CURRENT_BINARY_DIR}/grpc_build/gens/src/proto/
      ${GRPC_INCLUDE_DESTINATION}/src/proto/
  COMMAND
    ${RSYNC}
      --prune-empty-dirs --archive
      --include="*/" --include="*.h" --exclude="*"
      ${CMAKE_CURRENT_BINARY_DIR}/grpc_build/include/
      ${GRPC_INCLUDE_DESTINATION}/
  COMMAND
    ${RSYNC}
      --prune-empty-dirs --archive
      --include="*/" --include="*.h" --include="*.inc" --include="*.proto" --exclude="*"
      ${CMAKE_CURRENT_BINARY_DIR}/grpc_build/third_party/protobuf/src/
      ${GRPC_INCLUDE_DESTINATION}/
)

additionally , the source_path is not a ros package, how to understand the process git submodule update --init?

Shengye Wang · Answer 5 · Wed Aug 30 2023 19:12:18 GMT+0800 (China Standard Time)

Ah, I see the challenge here -- the script assumes a relative SOURCE_DIR, but you attempted to override it.

I suggest you try an alternative download option listed in https://cmake.org/cmake/help/latest/module/ExternalProject.html

How about

ExternalProject_Add(
  build_grpc
  PREFIX grpc
  URL /my/downloaded/grpc
  SOURCE_DIR grpc_build
  CONFIGURE_COMMAND ""
  BUILD_IN_SOURCE 1
  BUILD_COMMAND $(MAKE) CONFIG=opt EMBED_ZLIB=true
  INSTALL_COMMAND ""
)

or

ExternalProject_Add(
  build_grpc
  PREFIX grpc
  GIT_REPOSITORY /my/local/git_checkout/grpc/grpc.git
  GIT_TAG v1.34.1
  GIT_SHALLOW 1
  SOURCE_DIR grpc_build
  CONFIGURE_COMMAND ""
  BUILD_IN_SOURCE 1
  BUILD_COMMAND $(MAKE) CONFIG=opt EMBED_ZLIB=true
  INSTALL_COMMAND ""
)

(remember to run git submodule update --init --recursive in the local clone of grpc directory)

Tianyou Yu · Answer 6 · Thu Aug 31 2023 10:09:54 GMT+0800 (China Standard Time)

Appreciate your kind assistance, the problem are solved.

Here's the solution I used:

download grpc source in a local path, for example, /local/grpc
cd /local/grpc
git submodule update --init --recursive（ download submodule）
Modified the ExternalProject_Add part in the CMakeList.txt of the catkin_grpc as follows:

ExternalProject_Add(
  build_grpc
  PREFIX grpc
  URL /local/grpc
  SOURCE_DIR grpc_build
  CONFIGURE_COMMAND ""
  BUILD_IN_SOURCE 1
  BUILD_COMMAND $(MAKE) CONFIG=opt EMBED_ZLIB=true
  INSTALL_COMMAND ""
)

and then， cd ros package to build.

Thank you very much for helping me solve this problem.

Xuanrrr · Answer 7 · Wed Oct 18 2023 17:47:38 GMT+0800 (China Standard Time)

@9woods123
Hello, I followed your instructions but an error occurred. I'd appreciate it if you can offer some help.

Tianyou Yu · Answer 8 · Mon Oct 23 2023 13:59:51 GMT+0800 (China Standard Time)

@Xuanrrr

Hello, please provide more detail about what you have done and error info occured.
According to your error info, I guesses you are trying to extract an zip in a wrong path.
Please download your grpc source in a local path, for example, /local/grpc (source files instead of zip)

Tianyou Yu · Answer 9 · Sun Sep 22 2024 15:54:24 GMT+0800 (China Standard Time)

@9woods123 Hello, I followed your instructions but an error occurred. I'd appreciate it if you can offer some help.

you can ref to https://github.com/9woods123/catkin_grpc