PyPI package requires gazebo4, but ignition-citadel only ships with v3.7
FirefoxMetzger opened this issue · comments
I've installed ignition Citadel (LTS) from binaries following the instructions on Ubuntu Focal and then added gym-ignition from PyPI (pip install gym-ignition).
When I try to import the package I get the following stack trace
>>> import gym_ignition
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.8/dist-packages/gym_ignition/__init__.py", line 12, in <module>
from gym_ignition.utils import resource_finder
File "/usr/local/lib/python3.8/dist-packages/gym_ignition/utils/__init__.py", line 5, in <module>
from . import math
File "/usr/local/lib/python3.8/dist-packages/gym_ignition/utils/math.py", line 8, in <module>
from scenario import gazebo as scenario
File "/usr/local/lib/python3.8/dist-packages/scenario/__init__.py", line 130, in <module>
from .bindings import core
File "/usr/local/lib/python3.8/dist-packages/scenario/bindings/core.py", line 12, in <module>
import scenario.bindings.gazebo
File "/usr/local/lib/python3.8/dist-packages/scenario/bindings/gazebo.py", line 13, in <module>
from . import _gazebo
ImportError: libignition-gazebo4.so.4: cannot open shared object file: No such file or directoryWhich - I assume - happens because Ignition Gazebo is on v3.7 in Citadel, but v4.X is required by gym-ignition.
Currently, my aim is to write code on top of Ignition, so choosing the LTS seemed appropriate. Is there an LTS compatible version of gym-ignition, or is my best course of action to upgrade to Dome?
Update: Upgrading to Dome indeed resolves this error as expected.
Nonetheless, I'm curious if there is an option for Citadel.
Hi @FirefoxMetzger, thanks for reporting! The behavior you described is what expected. I try to argument a bit more the rationale behind this choice. You can find the official system requirements in Support Policy.
Ignition Gazebo is a brand new simulator and has not much in common with Gazebo Classic. The new simulator did not yet reach feature parity with Gazebo Classic, and development is moving pretty quickly. For us, maintaining old versions of the new simulator is very demanding, and we plan to start supporting an LTS version as soon as it will include all the features we desire. Unfortunately, Citadel is not yet complete.
If you really need it, we have an undocumented (on purpose) way to support Citadel, and this would require building the wheel locally in your PC. This method today should work because we do not yet use any new API from Dome (and, from the next gym-ignition release, Edifice). However, be aware that this undocumented way could get broken without notice at any time.
Right now I don't have a machine with Citadel. If you really want to proceed with Citadel, you can try the following, I will follow up if further errors appear.
pip install --global-option="build_ext" --global-option="-DIGNITION_DISTRIBUTION=Citadel" git+https://github.com/robotology/gym-ignition.gitNote that this is not tested in our CI pipeline, so I cannot guarantee that it is still functional, the last time I tried was few months ago. Note also that the command above builds in an isolated Python environment all the dependencies of gym-ignition from their sdist, therefore your system must also have all the build dependencies not only of gym-ignition but also all its python dependencies (e.g. idyntree). This is to say that it could take a while.
I'm trying on my system and I'm not sure that passing global options to pip works with recent versions. The most recent pip releases changed a lot and what reported above could be broken. The safe way is passing through the setup file:
git clone https://github.com/robotology/gym-ignition
cd gym-ignition
python setup.py install build_ext -D"IGNITION_DISTRIBUTION=Citadel"It should also provide a more verbose output that could be useful in case of problems.
Thanks for the detailed answer; I wasn't aware of this support policy.
An improvement that came to mind (I can do a PR if you wish), is that it could be nice to explicitly state that gym-ignition targets only the latest ignition release and update when it actually supports an LTS + latest release. Perhaps changing a line in the setup section of the Readme.md to Install Ignition Dome following the official instructions. could be of use as well. I also think the same for the statement on supporting all major OS'es in the support policy: explicitly state that currently only Linux is supported until others are and then update the documentation.
I get the feature completeness argument, and it makes sense to focus developmental effort on the latest ignition version. At the same time, a nice addition could be to add a tag to the last supported commit for each ignition release or LTS release and a statement along the lines of While we provide tags for each (LTS) version of Ignition, we only provide support for the latest release.. Potentially this could be paired with PyPI packages ala gym-ignition-citadel or gym-ignition-dome that build and release those tags. They don't need to be supported beyond mere existence, and it would tremendously help combat the version bingo that Ignition seems to be playing. 😃
Thanks for the suggestions. Having the same information scattered in many places increases the risk to forget to update some of them in the moment something changes. We recently moved a lot on information from the readme to new website, but it's far to be comprehensive enough. To address your points, I think it makes sense to add a link to the Support Policy webpage in the Setup section of the README. From there the current OSs support and the Ignition Gazebo versions should be clear.
I get the feature completeness argument, and it makes sense to focus developmental effort on the latest ignition version. At the same time, a nice addition could be to add a tag to the last supported commit for each ignition release or LTS release and a statement along the lines of While we provide tags for each (LTS) version of Ignition, we only provide support for the latest release.. Potentially this could be paired with PyPI packages ala
gym-ignition-citadelorgym-ignition-domethat build and release those tags. They don't need to be supported beyond mere existence, and it would tremendously help combat the version bingo that Ignition seems to be playing. 😃
Thanks, we will discuss some of these suggestions internally. At the moment, for every gym-ignition release, users can find the supported version of Ignition Gazebo in Releases. I know it's a bit hidden, we'll try to think of a more visible place.
As you noted, the release scheme of Ignition could confuse new users. I personally like their approach, particularly for 1) fast release cycle and 2) possibility to install side-by-side different major versions. I think that, particularly thanks to 1), we should not be concerned of using the most recent release. Of course, this backfires on us downstream users when it's time to select a version. Despite we started having some experience of distributing C++/Python packages, I still believe that gym-ignition is the project where we faced more challenges, and I'm not yet happy of the current situation. In our organization, we are recently exploring conda and conda-forge. Ignition Gazebo is not yet fully packaged there, but I personally believe that such system, due to all the limitations of the default Python packaging, is the only real way to address once and for all the version bingo problem you mentioned (and get multi-platform support). Stay tuned for further updates 😉
I quickly extend the previous comment saying that the v1.0beta2 Biro is the only release that officially supports Citadel. However, in our final v1.0 we considerably changed (and stabilized) the APIs. At this stage of development, for all those reading this issue in the future, I warmly recommend to avoid using old releases and, if really in need, try to follow what I wrote in #293 (comment).
In our organization, we are recently exploring conda and conda-forge. Ignition Gazebo is not yet fully packaged there, but I personally believe that such system, due to all the limitations of the default Python packaging, is the only real way to address once and for all the version bingo problem you mentioned (and get multi-platform support). Stay tuned for further updates 😉
I totally agree with @deigoferigo, in particular the feature that could be used in this context is "variants" https://conda.io/projects/conda-build/en/latest/resources/variants.html . A future release of gym-ignition on conda-forge could be released against two (or more) different Ignition releases, and then the solver in conda/mamba can install the preferred version based on the request of the user.
Conda Forge is indeed more versatile than PyPI and it is not hard to see how this is useful. At the same time, it forces an Anaconda dependency, which seems like an overkill when using containerization.
I use Docker+compose locally; this isn't officially supported (I think?) but makes managing C dependencies significantly easier and seems to work well enough in practice. It's also - so far - the only way I managed to get a GUI on Windows.
On the other hand, I know from experience that many people in academia (mostly students, but also PhDs) opt for a more self-destructive workflow where they "try things bear-metal until it works" and then get stuck with random error message due to incompatible dependencies of different projects that live on the same machine. For this use-case enforcing Anaconda might actually alleviate some pain, which is a really good thing.
Looking at the current topic, how does this solution sound: I'll submit a PR updating the Readme.md with a paragraph linking to a) the support policy, and b) @diegoferigo 's comment on building for previous versions. I'll also add a sentence on where to find the old releases for this case, but state that support will only be provided for the current version.
At the same time, it forces an Anaconda dependency, which seems like an overkill when using containerization.
Probably this is a bit OT, but just for your information: conda (package manager) != Anaconda (company and one quite rich and heavy installer that use conda) != conda-forge (community-driven distribution of software that uses conda as package manager, but that also ships extremely minimal distributions such as miniforge). A good recent blog post on using conda with Docker containers is https://uwekorn.com/2021/03/01/deploying-conda-environments-in-docker-how-to-do-it-right.html .
Having said that, I think there are several uses cases in which using apt or any other package manager over conda could make sense, I just don't know if size of the Docker image is one of those.
I use Docker+compose locally; this isn't officially supported (I think?) but makes managing C dependencies significantly easier and seems to work well enough in practice. It's also - so far - the only way I managed to get a GUI on Windows.
Just to understand when you say that Docker+composed is not "officially supported" to what do you refer? gym-ignition?
but that also ships extremely minimal distributions such as miniforge
Cool! I didn't know about miniforge. I'll have a closer look; thanks for the suggestion.
I just don't know if size of the Docker image is one of those
As far as my limited knowledge goes, it is likely not a factor for the current ignition. It would get more relevant once distributed simulations become stable because swapping nodes in a Kubernetes cluster means re-downloading the image, which increases container startup time, and reallocation can happen quite dynamically based on load.
Just to understand when you say that Docker+composed is not "officially supported" to what do you refer? gym-ignition?
I checked for an ign-gazebo docker image on the docker hub, but couldn't find one. I saw that both gym-ignition and ign-gazebo have Dockerfiles, but I didn't find related documentation (to be fair, I didn't search extensively). I also found diegoferigo/gym-ignition, which I'd assume is a build from said files. However, there was no associated description, and I didn't want to assume.
From this, I concluded that - while there seems to be support for docker - it doesn't seem like an officially intended way to interact with the libraries. Hence why I think it may exist, but that it may not be supported officially. That said, I could be very wrong with my assumption since I haven't read all of the available documentation for ign-gazebo and gym-ignition yet.
I just don't know if size of the Docker image is one of those
As far as my limited knowledge goes, it is likely not a factor for the current ignition. It would get more relevant once distributed simulations become stable because swapping nodes in a Kubernetes cluster means re-downloading the image, which increases container startup time, and reallocation can happen quite dynamically based on load.
Sorry, I was probably not clear: I totally understand the concern in having a small size for the Docker image (and indeed, not currently supporting musl libc is one reason for which someone could not want to use conda-forge), I just don't think that the size that you will get installing gym_ignition via apt + pip is substantially different from the one that you could get by installing (once and if available) gym_ignition via conda-forge.
I checked for an ign-gazebo docker image on the docker hub, but couldn't find one. [...]
Thanks for the clarification! I see, indeed no Docker official images are available (neither in https://hub.docker.com/u/osrf) but I think that nevertheless running the application or using the library in Docker is not any different then using it from any other Linux system, so (unless I am missing) anything that is supported on Linux is also supported on Docker, or I am missing something?
[...] anything that is supported on Linux is also supported on Docker, or I am missing something?
Yes and no. As long as the container's kernel is present on the host, yes. On Windows, the new recommended way to run Docker is via WSL2. Microsoft is using their own kernel for this, and there are some differences in terms of driver support. In particular, GPU acceleration is limited as it is/was intended as a console-only experience. The current dev-channel builds added support for CUDA (I think OpenCL, too?) and highly experimental support for hardware-accelerated rendering (which currently makes my Windows installation unstable).
Looking at the current topic, how does this solution sound: I'll submit a PR updating the Readme.md with a paragraph linking to a) the support policy, and b) @diegoferigo 's comment on building for previous versions. I'll also add a sentence on where to find the old releases for this case, but state that support will only be provided for the current version.
I'd be glad to review a PR for a). Building ScenarIO for previous version is and will remain an undocumented hack, reporting it in the README would mean that we suggest users following that path. I'd prefer to avoid giving expectations, as I wrote above, we keep the option to break Citadel support on the current gym-ignition version (that requires Dome) at any time (one example is the stale #249).
From this, I concluded that - while there seems to be support for docker - it doesn't seem like an officially intended way to interact with the libraries. Hence why I think it may exist, but that it may not be supported officially.
For what concern Docker support, Open Robotics started supporting Docker even though it is still preliminary. You can learn more in ignition-tooling/gzdev and this youtube video.
I saw that both gym-ignition and ign-gazebo have Dockerfiles, but I didn't find related documentation (to be fair, I didn't search extensively). I also found diegoferigo/gym-ignition, which I'd assume is a build from said files.
Those resources are mainly for CI, and they are not meant to be used by downstream users. However, if any user wants to dockerize their application, those resources could be an excellent starting point. I personally run my local development environment and my distributed trainings entirely in docker, but in my experience, particularly for distributed applications that could also need other challenging specs (GPU support, ML frameworks, ...), creating custom images from scratch is often the best solution. We never put effort to distribute Docker images for this reason.
This being said, I am not a Windows user nor I know substantial differences in the setup, and all wrote above refers to GNU/Linux hosts.
I get the feature completeness argument, and it makes sense to focus developmental effort on the latest ignition version. At the same time, a nice addition could be to add a tag to the last supported commit for each ignition release or LTS release and a statement along the lines of While we provide tags for each (LTS) version of Ignition, we only provide support for the latest release.. Potentially this could be paired with PyPI packages ala
gym-ignition-citadelorgym-ignition-domethat build and release those tags. They don't need to be supported beyond mere existence, and it would tremendously help combat the version bingo that Ignition seems to be playing. smileyThanks, we will discuss some of these suggestions internally. At the moment, for every gym-ignition release, users can find the supported version of Ignition Gazebo in Releases. I know it's a bit hidden, we'll try to think of a more visible place.
A quick update on this. We started using GitHub Discussions and beyond posting the new published releases in Releases, we will also open a new topic in Discussion/Releases.