Installing only certain packages

[Jérémie Dumas]

Hi there,

Thanks for this experimental CMake dependency manager. Conan is probably the only C++ package manager to offer such a feature at the moment.

One question I had trying to integrate is my project, is how do you propose to install only certain dependencies based on some package configuration options. Let's say I had a library that depends on A and B, but depending which CMake build option I selected, I only need to install A, or only need to install B. Currently it seems the first call to find_package() in CMake will trigger the call to conan install, and that will install all the dependencies in the conanfile.txt. Wouldn't it be better to install each required dependency one by one as calls to find_package() are being emitted in CMake? (if that's even possible, I'm new to Conan).

[James]

Hi @jdumas

Thanks for your question

Wouldn't it be better to install each required dependency one by one as calls to find_package() are being emitted in CMake? (if that's even possible, I'm new to Conan).

That is a good point. In general it is not possible, due the the transitive dependencies. If you have 2 different packages, like boost and opencv, which both depend on zlib, then they could be using different zlib versions. If you install them separately, this will not be known, and only when trying to use both libraries at the same time, problems might happen due to the different zlib versions. These problems also tend to be very challenging to debug, being often complex runtime errors. Installing all dependencies in the same graph ensures consistency of the transitive dependencies, or Conan will be raising "conflicts". Users can resolve those conflicts downstream, by defining which version is to be used in the transitive dependencies, and that can in turn fire a re-build of its dependants. In the example of opencv and boost depending on zlib, if the consumer forces a zlib version different to the ones that were used to build boost, then boost might require a rebuild from source against the forced zlib version.

Regarding the conditonality in the cmake-conan integration that is a bit more challenging. You can perfectly write a conanfile.py instead of a conanfile.txt and the cmake-conan integration will use it too. That conanfile.py can also contain conditional logic, have a look at: https://docs.conan.io/2/tutorial/consuming_packages/the_flexibility_of_conanfile_py.html. But still, it is not possible to fully align it with the runtime execution of CMake. But maybe if the conditions are simple enough, like depending on the os, then implementing those conditionals should be fairly simple.

If the conditionals logic is complex, or is simply based on user inputs, probably the way to move would be:

• Copy the conan_provider.cmake into the project (this is recommended in any case).
• Customize the conan_provider.cmake to add to the conan install call the -o myoption:myvalue values that you need
• Use the options in your conanfile.py requirements() method to conditionally self.requires() different dependencies.

Please let me know if this helps.

[Jérémie Dumas]

Thanks for the quick reply. I can see how this is difficult to handle in general. I am wondering if it's possible to decouple "resolving" the dependency version (given all the potential deps listed in the conanfile.txt) from the installation part? But then, at the same time, if we want to have separate mutually-exclusive dependencies (e.g. TBB 2020 and OneTBB 2021), then it might not be possible to do that in general.

Anyway thanks for the suggestions. I will learn more the conanfile.py and see if I can make this work. At the very least I'd like to have a set of CMake options I can use to enable various components and forward them to the conanfile.py to generate the appropriate global requirements.

Feel free to close this issue!

[James]

Thanks for the quick reply. I can see how this is difficult to handle in general. I am wondering if it's possible to decouple "resolving" the dependency version (given all the potential deps listed in the conanfile.txt) from the installation part? But then, at the same time, if we want to have separate mutually-exclusive dependencies (e.g. TBB 2020 and OneTBB 2021), then it might not be possible to do that in general.

Conan actually does the installation of dependencies in 3 stages: The first stage is compute the "versions" graph, which means fetching the recipes for those dependencies and transitive dependencies. Then in the second stage computes the "package_id" of each node in the graph, as the result of several factors, and finally the packages are installed.

Still the first stage is the one that has to be evaluated with all the necessary requirements, not one at a time. The dependency graph can be quite complex, there are conditional requirements, overrides, version-ranges, etc. Evaluating it can be complex and costly, because evaluating each hypothesis in the graph is expensive, even requiring fetching files from servers, parsing and executing recipes, etc. This is the stage that detects dependencies versions conflicts, much earlier than actually trying to install the packages.

Anyway thanks for the suggestions. I will learn more the conanfile.py and see if I can make this work. At the very least I'd like to have a set of CMake options I can use to enable various components and forward them to the conanfile.py to generate the appropriate global requirements.

Feel free to close this issue!

Sounds good. I'll close the issue by now, but please do not hesitate to open new tickets for any further question you might have. Thanks for the feedback!

[Jérémie Dumas]

Conan actually does the installation of dependencies in 3 stages: The first stage is compute the "versions" graph, which means fetching the recipes for those dependencies and transitive dependencies. Then in the second stage computes the "package_id" of each node in the graph, as the result of several factors, and finally the packages are installed.

I see, that's really helpful thanks. Is it possible to call these steps separately? I see that there is an experimental Python API that supports installing packages for a specific dependency graph, but it doesn't seem possible to install dependencies for a specific subgraph? I guess it's not super useful other than saving space on disk, but on the CMake side you'd be able to tell if a find_package() for an optional dependency fails.

[James]

I see, that's really helpful thanks. Is it possible to call these steps separately? I see that there is an experimental Python API that supports installing packages for a specific dependency graph, but it doesn't seem possible to install dependencies for a specific subgraph?

Yes, it is possible via the PythonAPI. But we would be talking about an approach wildly different of using the cmake-conan integration. If CMake is driving the installation of dependencies, then trying to access the sub-api would be quite challenging, and in any case overkill. If we are talking about this complexity, this approach is definitely not worth it, because a conan install . + cmake --preset conan-default will make it much better.

And still, even if it was possible to call the installation of subgraphs, this wouldn't require the PythonAPI, this can already be implemented via regular conan install --requires=xxx commands, and it could be implemented in the conan_provider.cmake. But the issue of inconsistent and incompatible dependencies would be an issue. Maybe I didn't understand your comment correctly?

[Jérémie Dumas]

this can already be implemented via regular conan install --requires=xxx commands, and it could be implemented in the conan_provider.cmake

That sounds good actually, it could do what I want. I'm still very new to Conan, so I'll need to spend more time experimenting to figure out what I want exactly, thanks for bearing with me!

[James]

That sounds good actually, it could do what I want. I'm still very new to Conan, so I'll need to spend more time experimenting to figure out what I want exactly, thanks for bearing with me!

No, please, trying to install one dependency at a time, for each find_package() will inevitably lead sooner or later to issues and problems, that will require way more time to debug and understand. The fact that is possible to implement it via CMake provider doesn't make it less problematic. I'd strongly advise against that 😅

[Jérémie Dumas]

Ok ok! I thought it would still compute the graph & version from the whole conanfile, and only do the install of a specific package.