######################################################################## ######################################################################## ## USGS ISIS CMake build system notes ## ######################################################################## ######################################################################## #============================================================================== #============================ Introduction ==================================== #============================================================================== # Overview -------------------------------------------------------------- This build system was developed for the post-3.5.0 version of the USGS ISIS software. The intent is to fully replace the existing Makefile based build system with something that is easier to update and maintain. # Requirements -------------------------------------------------------------- - Tested Operating Systems Ubuntu 14, CentOS 7, and OSX 10. - Tested Compilers Linux: GCC 4.8.4 OSX: Clang 7.3.0 - Other software: CMake version 3.2 or higher. The 3rd party library requirements for ISIS are the same as they are with the old Makefile build system. # CMake information ------------------------------------------------------------ There is not a really good CMake guide on the web but here are some useful resources for getting started with it: The official CMake tutorial --> https://cmake.org/cmake-tutorial/ A CMake syntax overview --> https://cmake.org/Wiki/CMake/Language_Syntax A list of all the supported CMake commands and their arguments --> https://cmake.org/cmake/help/v3.2/manual/cmake-commands.7.html A list of standard CMake variables that may be used --> https://cmake.org/Wiki/CMake_Useful_Variable #============================================================================== #========================= Using the build system ============================= #============================================================================== # Running the build -------------------------------------------------------- Using the new build system is very similar to building other CMake based software. Start by creating a new build folder, a common choice is a folder named "build" in the same folder as the source folder. By using a new folder you can easily delete it to get a completely clean build and you won't introduce any new files into your source folder. Change directory into the new folder, then run cmake pointed at the top level of the ISIS folder. ISIS supports the following special options: isis3Data = Specify where the downloadable ISIS data is located. This folder contains mission kernels and sensor info. isis3TestData = Specify the location containing the special ISIS application test data. This folder contains input and truth folder for application and module tests. testOutputDir = Specify where output folder from running app/module tests will go. Can be set to the same location as isis3TestData. buildCore = Set to OFF to skip building core libraries. This includes libisis3 and the "plugin" libraries such as libShade, libTopo, and libPlanar. buildMissions = Set to OFF to skip building mission code. This includes all the libraries with mission names such as libMsiCamera and libcassini. buildStatic = Set to ON to build the core library as static in addition to dynamic. Adding a static build will increase the total build time. buildTests = Set to OFF to disable building all unit tests. If this is set to ON, all tests will be built for all enabled modules. You can also use standard CMake options. The following options will probably be the most frequently used: CMAKE_INSTALL_PREFIX = Specify where the installation folder. CMAKE_BUILD_TYPE = Choose Debug, Release, RelWithDebInfo, or MinSizeRel Once CMake finishes running, run "make" to build the code, "make install" to install it to the output, and "make docs" to generate the documentation files in the installation folder. Use the "-j" argument to specify the number of threads to use. A full set of commands may look like: cd ~/isis_code mkdir build cd build cmake -DCMAKE_INSTALL_PREFIX=~/isis_install -Disis3Data=~/isis3data \ -Disis3TestData=~/isis3TestData ~/path/to/isis/source make -j 4 make install -j 4 make docs The build system also accepts the "make clean_source" command which removes all files and folders that need to be created in the top level of the ISIS folder in order to run the unit tests. # Building individual components -------------------------------------------- Unfortunately the design of CMake does not lend itself to building individual software components independently of each other but it is still possible with some effort. The trick to doing this is that the output of CMake is a Makefile based system and those Makefiles can be accessed directly. For example, to make a change to the file src/base/objs/Pvl/Pvl.cpp and rerun a unit test, the following steps can be performed: 1 - Start from the build folder. 2 - Recompile the .o file with this command: >> make -f src/CMakeFiles/isis3.dir/build.make src/CMakeFiles/isis3.dir/base/objs/Pvl/Pvl.cpp.o 3 - Recompile the unit test with this command: >> make isis3_unit_test_Pvl 4 - Run the unit test normally. >> ctest -R isis3_unit_test_Pvl If one of the make commands has no effect, try manually deleting the existing file you are trying to build and then re-run the make command. # Running tests ------------------------------------------------------------ The CMake build system relies on the CTest tool, which should be included with CMake, to handle unit, application, and module tests. To run the tests, simply run the ctest command from the build directory. The CTest tool has a number of useful built-in options to help run the tests which you can see by running "ctest --help". Note that due to the naming convention applied to the tests in the build system you can choose to run only a certain category of tests by adding the options "-R _unit_", "-R _app_", or "-R _module_". Application and module test output folders will be placed in a directory structure underneath the folder specified with the "testOutputDir" option. Console output from running tests is placed in the [build_folder]/src location in files ending with the extension ".output" and with names matching the test they correspond to. These test files are only retained if the test failed. You can find additional information about the last test that was run by looking at the file [build_folder]/Testing/Temporary/LastTest.log. # Building the documentation -------------------------------------------------------- To generate the documentation, just run "make docs" after running CMake. All of the relevant documentation should be placed in the installation folder. Intermediate documentation build files will be placed in the build folder, but the final output documentation will only be created in the install folder. The third party tools "xalan", "doxygen", and "latex" must be available in order to generate the documentation. # Build versus install versus source ------------------------------------------------ Unless you specify the source directory as the build or test output directory, very few files will be placed in the source folder as part of the build. All of the other files generated by running "make" will be placed in the build folder. The build folder contains all of the compiled .o files, generated protobuf/uic/moc files, libraries, and executables. The build folder also contains many automatically generated CMake files which can usually be ignored. The "make install" step is only required to prepare software for distribution. The installation folder will contain only the libraries, binaries, supporting files, and documentation that a user needs to run ISIS tools. Source code, tests, and build system files are not placed in to the installation folder. The installation folder should be ready to tar up and be used on another computer. # Cleaning up the build ------------------------------------------------------------- To completely clean up the build just delete the entire build folder and create a new one. You can also run "make clean" to remove files generated during the build as opposed to files generated while running CMake. To remove all cmake generated files from the source directory (there should only be a couple folders) run "make clean_source". # Troubleshooting ------------------------------------------------------------- Q: I am running cmake from a build directory but the files are being generated in the source directory. A: Completely clean the cmake generated files from the source directory and try again. If generated cmake files are there it will continue to generate them in that location. Q: The build fails early on complaining about a library not found. A: Make sure that the dynamic libraries needed to run uic, moc, and protoc are in the search path. This may mean adding the 3rdParty/lib folder to LD_LIBRARY_PATH. Q: A third party library is not being found. A: Verify that the library is installed on your system or in the 3rdParty/lib folder. The build system does not search for libraries outside these locations. #============================================================================== #======================== Build system details ================================ #============================================================================== # File listing ---------------------------------------------------------------- The CMake build system consists of 15 files, each of which is described below. README.txt = This file! Contains a description of the build system. CMakeLists.txt = The top level build file. Handles input options, defines project characteristics, sets up some required links, and calls all of the other required modules. src/CMakeLists.txt = The lower level build file. This defines which module folders are used to create libraries. There is one core library and one library per mission directory. The source files under this folder are found by search functions running from this folder. In the /cmake folder: AddIsisModule.cmake = Functions for parsing a module folder and adding all of the components to the build. BuildDocs.cmake = This file handles tho documentation production. CodeGeneration.cmake = Functions for generating moc, uic, and protobuf files. FindAllDependencies.cmake = Identify all of the required 3rd party libraries. InstallThirdParty.cmake = Handle transfer of 3rd party libraries to the install folder. RunMakeFileTest.cmake = Script that is called to execute an app or module (Makefile based) test. RunUnitTest.cmake = Script that is called to execute a unit test. TestSetup.cmake = Helper functions for setting up tests. Utilities.cmake = Miscellaneous functions used in the other files. In the /scripts folder: finalizeInstalledOsxRpaths.py = Script to correct the RPaths of OSX libraries after they are installed. IsisInlineDocumentBuild_mod.xsl = An old documentation build file modified for use by the CMake build system. fetchRequiredData.py = Script for fetching a subset of the ISIS mission data for the purpose of running unit tests. Not required by the build system. # Maintenance tips ---------------------------------------------------------------- - There are a number of file requirements of ISIS that the build system takes extra steps to work around. These include: - Unit tests (as opposed to application or module tests) must be run from an executable named "unitTest" in order for certain GUIs to be suppressed. They must have the associated xml file with the same name in the same directory. - Application xml files must be located in $ISISROOT/bin/xml - $ISISROOT/IsisPreferences must exist. - $ISISROOT/src/base/objs/Preference/TestPreferences must exist to run tests. - New files are added automatically when CMake is run, but new module folders must be manually added in src/CMakeLists.txt. Tests of all types are added automatically. - Required 3rd party libraries are explicitly listed in FindAllDependencies.cmake - Miscellaneous installation and build settings are found in the top level CMakeLists.txt - Turning on static building of the core library increases the build time. - The file organization in the build folder does not mimic the file organization in the source directory but all of the .o files are there and can be accessed. # Possible new features ----------------------------------------------------------- - Building/testing individual modules from within the CMake build directory is inconvenient. It would be possible to add any number of separate build targets so that a user could type "make base_Pvl_test" or something similar from the build folder and have the relevant class and its unit tests rebuilt. - It is easy in CMake to add new build targets to do things like "remove all output folders from the test output directory". # Future development ---------------------------------------------------------------- - libisis is the only library with a static build option but it would be easy to make the other libraries also built as static. - CMake has its own system for unit test coverage that could be used as an alternative for a proprietary system. The current CMake build system does not provide any test coverage options. - CMake has tools for automated fetching and building of 3rd party libraries that could be used to obtain all of the ISIS dependencies. Ames Stereo Pipeline has a longstanding system to do this that installs nearly all of ISIS' dependencies. - Unlike in the MakeFile build, there are not CMake build files spread throughout the source tree. Setting up a tree of CMakeLists.txt files should make the CMake generated build files more closely resemble the source directory but it would require the creation of many more files (as of now there are almost 2400 Makefiles in the src folder.) Some special handling may be needed to make this work with the mutually interdependent core library folders.