conda_deps packages being overwritten with PyPi equivalents.

[David Banas]

I have the following tox.ini file:

[tox]
envlist = py38, py39, py310, pylint, format, flake8, docs
skip_missing_interpreters = true
isolated_build = true

[testenv]
deps =
    pytest
    pytest-xdist
    pytest-cov
conda_deps =
    scikit-rf
    numpy
    scipy
conda_channels =
    defaults
    conda-forge
conda_install_args =
    --force-reinstall
    --override-channels
conda_create_args =
    --override-channels
setenv = CONDA_DLL_SEARCH_MODIFICATION_ENABLE=1
commands =
    py.test --basetemp={envtmpdir} -vv --cov=pybert \
            --cov-report=html --cov-report=term-missing tests
{snip}

But, after I run: tox -e py38 -r, I find that I have PyPi versions of: scikit-rf, numpy, and scipy, installed in the test environment:

(pybert-dev)
capnf@DESKTOP-G84ND7C MINGW64 ~/Documents/GitHub/PyBERT (add-tests)
$ conda activate .tox/py38/

(C:\Users\capnf\Documents\GitHub\PyBERT\.tox\py38)
capnf@DESKTOP-G84ND7C MINGW64 ~/Documents/GitHub/PyBERT (add-tests)
$ conda list numpy
# packages in environment at C:\Users\capnf\Documents\GitHub\PyBERT\.tox\py38:
#
# Name                    Version                   Build  Channel
numpy                     1.23.3                   pypi_0    pypi

(C:\Users\capnf\Documents\GitHub\PyBERT\.tox\py38)
capnf@DESKTOP-G84ND7C MINGW64 ~/Documents/GitHub/PyBERT (add-tests)
$ conda list scipy
# packages in environment at C:\Users\capnf\Documents\GitHub\PyBERT\.tox\py38:
#
# Name                    Version                   Build  Channel
scipy                     1.9.2                    pypi_0    pypi

(C:\Users\capnf\Documents\GitHub\PyBERT\.tox\py38)
capnf@DESKTOP-G84ND7C MINGW64 ~/Documents/GitHub/PyBERT (add-tests)
$ conda list scikit-rf
# packages in environment at C:\Users\capnf\Documents\GitHub\PyBERT\.tox\py38:
#
# Name                    Version                   Build  Channel
scikit-rf                 0.24.1                   pypi_0    pypi

I see my conda_deps in the log:

py38 recreate: C:\Users\capnf\Documents\GitHub\PyBERT\.tox\py38
py38 installcondadeps: scikit-rf, numpy, scipy
py38 installdeps: pytest, pytest-xdist, pytest-cov
py38 inst: C:\Users\capnf\Documents\GitHub\PyBERT\.tox\.tmp\package\1\PyBERT-4.0.0.tar.gz

Why are they being overwritten with Pip / PyPi equivalents?

[Antoine Dechaume]

This is because you have pinned the version of those dependencies in pyproject.toml and if these are not satisfied when conda installs them from conda_deps then they are overridden by pip when your package is installed.

[David Banas]

This is because you have pinned the version of those dependencies in pyproject.toml and if these are not satisfied when conda installs them from conda_deps then they are overridden by pip when your package is installed.

Thank you, @AntoineD !
Is there anywhere in the log where I'm alerted that conda failed to install the requested versions and that pip will be saving the day?

[Antoine Dechaume]

conda ignores the dependencies defined in packaging descriptions for pip like pyproject.toml, you may see that those dependencies have been installed twice by adding -v to the tox command line.

If you intend to create a pip installable package , you do not need conda here.
If you need to use conda anyway, then you may use the very same version pinning in tox.ini as they appear in pyproject.toml.

[David Banas]

Thanks, @AntoineD !

Using a freshly deleted and recreated .tox/py38/ environment, I tried the following experiment, which I expected to fail:

$ conda install -c defaults -c conda-forge --override-channels python=3.8 numpy==1.23.3 scipy==1.9.2 scikit-rf==0.24.1

But, it succeeded!

So, I continued on with: tox -e py38. (Note the missing -r option.):

$ tox -e py38
{snip}
py38 create: C:\Users\capnf\Documents\GitHub\PyBERT\.tox\py38
py38 installcondadeps: scikit-rf, numpy, scipy
py38 installdeps: pytest, pytest-xdist, pytest-cov
py38 inst: C:\Users\capnf\Documents\GitHub\PyBERT\.tox\.tmp\package\1\PyBERT-4.0.0.tar.gz

Note the missing --no-deps in the last command above; significant?

I'm finding that my conda-installed numpy package is still getting overwritten:

$ conda activate .tox/py38/

(C:\Users\capnf\Documents\GitHub\PyBERT\.tox\py38)
capnf@DESKTOP-G84ND7C MINGW64 ~/Documents/GitHub/PyBERT (add-tests)
$ conda list numpy
# packages in environment at C:\Users\capnf\Documents\GitHub\PyBERT\.tox\py38:
#
# Name                    Version                   Build  Channel
numpy                     1.23.3                   pypi_0    pypi

And I'm wondering if it's the installation of the PyBERT package that is causing the overwrite.
(Note the missing --no-deps option in the py38 inst: C:\...\PyBERT...tar.gz command above.)

I tried repeating the manual conda install, but that fails:

(C:\Users\capnf\Documents\GitHub\PyBERT\.tox\py38)
capnf@DESKTOP-G84ND7C MINGW64 ~/Documents/GitHub/PyBERT (add-tests)
$ conda install -c defaults -c conda-forge --override-channels python=3.8 numpy==1.23.3 scipy==1.9.2 scikit-rf==0.24.1 --force-reinstall
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working...
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:

  - defaults/win-64::bottleneck==1.3.5=py38h080aedc_0
  - defaults/win-64::matplotlib==3.5.3=py38haa95532_0
  - defaults/win-64::mkl_fft==1.3.1=py38h277e83a_0
  - defaults/win-64::mkl_random==1.2.2=py38hf11a4ad_0
  - defaults/win-64::numexpr==2.8.4=py38h5b0cc5e_0
  - defaults/win-64::numpy==1.23.4=py38h3b20f71_0
  - defaults/win-64::pandas==1.5.1=py38hf11a4ad_0
done

[David Banas]

If you need to use conda anyway, then you may use the very same version pinning in tox.ini as they appear in pyproject.toml.

That worked!
Thank you. :)