Question about environment.yml

[Anderson Bravalheri]

Hello @FlorianWilhelm, thank you very much for the handcrafted template! Very useful specially for people that still don't have a preferred workflow for dealing with data, like me.

After generating a new project with pyscaffoldext-dsproject I was wondering what would be the advantages of having all the requirements listed in the environment.yml instead of installing the project itself in editable mode...

For example, what would be the disadvantages of doing:

@@src/pyscaffoldext/dsproject/templates/environment_yml.template:23@@
  - pip:
     # add here only pip-packages that are not available in conda/conda-forge! E.g.:
     - optuna
+    - -e .[testing]
+    - -r docs/requirments.txt
  # for development only (could also be kept in a separate environment file)
  - jupyterlab
- - pytest
- - pytest-cov
  - tox
  - pre_commit
  - nbdime
  - nbstripout
-  - sphinx
-  - recommonmark

[Florian Wilhelm]

Interesting question! I have never seen - -e .[testing] within the pip: section of an environment.yml. So it installs the package itself in editable mode and one no longer needs to call python setup.py develop manually? Nice! What does the [testing] do?

The thing is that conda packages are often quite different than wheel packages. Installing certain packages with pip within a conda environment might even break things in a very subtle way. py.test might be one of them since it uses virtualenv. Conda has a modified version of virtualenv that works nicely together with conda. If you install it with pip it might break out of your conda environment. Thus the rule of thumb is to always use conda to install a package and only resort to pip if the package is not available in conda (also conda-forge).

[Anderson Bravalheri]

I see... Thank you very much for the reply @FlorianWilhelm.

The .[testing] part install the package along side with the options.extra_require > testing dependencies listed in setup.cfg. (In the case of PyScaffold generated projects that would be setuptools, pytest and pytest-cov). Not that it is really required if someone is using tox for testing (our tox.ini is configured to install automatically the dependencies listed in options.extra_require > testing).

The second part of your answer made me think: would it mean that basically we cannot use tox in this circumstances since it would attempt to a) create a virtualenv and b) install the package and its dependencies in this virtualenv using pip? I know that there is a tox-conda, but it seems to only solve (a), not (b).

[Anderson Bravalheri]

I suppose if one install the conda's version of tox, it is also patched to use the patched virtualenv and therefore works nicely within conda, but it also means that the dependencies and extra test packages being downloaded by tox come from PyPI anyway.

[Florian Wilhelm]

Yes, exactly. Everything works as long as you use conda versions of special packages like virtualenv, ipython, etc. and since tox then uses the patched version of virtualenv everything works as expected. This is why I added those packages to environment.yml.

[Anderson Bravalheri]

Thank you very much for the explanation @FlorianWilhelm