conda-lock fails to respect non-default git branch/ref/rev from poetry `pyproject.toml`

[Cameron Smith]

Checklist

• I added a descriptive title
• I searched open reports and couldn't find a duplicate

What happened?

On

❯ conda-lock --version  
conda-lock, version 2.5.2.dev12+g0481cc6

if I specify any

ONE of branch or rev

[tool.poetry.dependencies]
# Other dependencies...

flytekit = { git = "https://github.com/cameronraysmith/flytekit.git", rev = "b1e64ba20f1ddd83723e455d2bf893d8f3fc84b7" }
flytekit = { git = "https://github.com/cameronraysmith/flytekit.git", branch = "3928-pandas-2" }

other than the default branch HEAD in the dependencies and run conda-lock

❯ poe -d conda-lock
Poe => conda-lock --conda mamba --kind lock --kind env --no-dev-dependencies --filter-categories --category workflows --virtual-package-spec environments/conda/virtual-packages.yml --filename-template 'environments/conda/conda-{platform}.lock' --lockfile environments/conda/conda-lock.yml -f pyproject.toml

with virtual-packages.yml,

subdirs:
  linux-64:
    packages:
      __cuda: 12.3.0
      __glibc: 2.35

then

• for the case of specifying a commit sha as rev, I get an error stating that conda-lock cannot checkout the default branch even though the commit sha is not part of that branch

    raise CalledProcessError(
conda_lock._vendor.poetry.utils._compat.CalledProcessError: Command '['git', '--git-dir', '/var/folders/zx/66kfpxbj16n3blcbnn1kys9h0000gp/T/pypoetry-git-flytekitj9ytruxm/.git', '--work-tree', '/var/folders/zx/66kfpxbj16n3blcbnn1kys9h0000gp/T/pypoetry-git-flytekitj9ytruxm', 'checkout', 'master']' returned non-zero exit status 1.

• or, if I specify a non-default branch, the commit sha that is placed in the conda-lock output is the HEAD of the default branch having ignored the constraint pointing to a non-default branch or sha that is not part of the default branch history.

This also holds true if I place all the dependencies

inside the conda-lock dependencies table.

[tool.conda-lock.dependencies]
flytekit = { git = "https://github.com/cameronraysmith/flytekit.git", branch = "3928-pandas-2" }

In all cases, the

correct sha appears in the poetry.lock file.

[package.source]
type = "git"
url = "https://github.com/cameronraysmith/flytekit.git"
reference = "3928-pandas-2"
resolved_reference = "b1e64ba20f1ddd83723e455d2bf893d8f3fc84b7"

Conda Info

conda info

mamba version : 1.5.3
     active environment : base
    active env location : /usr/local/Caskroom/mambaforge/base
            shell level : 1
       user config file : /Users/user/.condarc
 populated config files : /usr/local/Caskroom/mambaforge/base/.condarc
                          /Users/user/.condarc
          conda version : 23.10.0
    conda-build version : not installed
         python version : 3.10.8.final.0
       virtual packages : __archspec=1=m1
                          __osx=13.6=0
                          __unix=0=0
       base environment : /usr/local/Caskroom/mambaforge/base  (writable)
      conda av data dir : /usr/local/Caskroom/mambaforge/base/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/bioconda/osx-64
                          https://conda.anaconda.org/bioconda/noarch
                          https://conda.anaconda.org/conda-forge/osx-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/nodefaults/osx-64
                          https://conda.anaconda.org/nodefaults/noarch
          package cache : /usr/local/Caskroom/mambaforge/base/pkgs
                          /Users/user/.conda/pkgs
       envs directories : /usr/local/Caskroom/mambaforge/base/envs
                          /Users/user/.conda/envs
               platform : osx-64
             user-agent : conda/23.10.0 requests/2.28.1 CPython/3.10.8 Darwin/22.6.0 OSX/13.6 solver/libmamba conda-libmamba-solver/23.11.0 libmambapy/1.5.3
                UID:GID : 502:20
             netrc file : None
           offline mode : False

Conda Config

conda channels from pyproject.toml `tool.conda-lock` table

these are the channels specified in pyproject.toml

[tool.conda-lock]
channels = [
    'nodefaults', 
    'pytorch',
    'nvidia',
    'conda-forge', 
    'bioconda', 
]

Conda list

I'm not working in an environment. I'm just using conda-lock from a pipx installation to generate conda-lock and environment files.

[Ben Mares]

This may be due to our use of an older vendored version of Poetry. I hope to get a chance soon to upgrade it, but there is no timeline for that.

[Cameron Smith]

Thanks Ben!

[Cameron Smith]

It's certainly straightforward to patch this manually, at least up to the content hash of the lock file itself. Is the latter straightforward as well? I will update this comment if I find this in the source or docs.

[Ben Mares]

It's certainly straightforward to patch this manually, at least up to the content hash of the lock file itself.

I'm not following, what do you mean specifically?

Is the latter straightforward as well?

I'm guessing that revendoring Poetry would be a day-long project for me, if that's what you're asking. It's really tough for me to find a full day to dedicate to this.

I will update this comment if I find this in the source or docs.

Any additional info would be much appreciated. Thanks so much for taking the time to report this!

[Cameron Smith]

Oh I see. I should have been more explicit. I was speaking from the perspective of user workaround. I did not mean to comment on the complexity of fixing the root cause of the issue and I certainly do appreciate the time investment that would be required to do so.

The workaround I was implicitly proposing was that a user can manually update the commit sha for the target dependency in the conda-lock file that incorrectly resolves to the default branch HEAD. After this, I believe the only issue remaining would be that the value of the metadata.content_hash field of the lock file would then be incorrect.

I believe this is not used, so it could potentially be left incorrect, but I might be wrong about that. In case it is required, I meant to ask what exactly the key for the metadata.content_hash is comprised of and if it could be manually recomputed without significant complication for a complete end-user workaround for this issue.

[Ben Mares]

The metadata.content_hash will hopefully be removed sometime soon, as per #432.

I'm not sure offhand how well your proposed fix of patching the lockfile would work. I'd recommend trying it out first by hand. I'm curious about the result.

[Cameron Smith]

The metadata.content_hash will hopefully be removed sometime soon, as per #432.

Many thanks for this pointer.

I'm not sure offhand how well your proposed fix of patching the lockfile would work. I'd recommend trying it out first by hand. I'm curious about the result.

I will comment on the results if I test this.