AssertionError in resolve module from not adding dependencies on strictness == 3
asmeurer opened this issue · comments
See conda/conda-build#205. When the strictness of a package is 3, the dependencies are not added. This can lead to issues if not all the dependencies of that package are specified. At the very least, the error message should be improved.
The strictness check was removed at 3077dc1 but reverted at c4c844b because it caused some serious performance regressions, because the larger number of packages created a much larger version equation with many more nonzero terms for the SAT solver.
For instance, for conda create -n test anaconda python=2.7 numpy=1.6 before this "fix" the solution was 146 and after it was 218.
Might be worth re-visiting this in the light of #1702
Agreed. I saw this when working on 1702. I do not understand the logic and think it should be fixed.
Another vote for a fix for this if possible- we have quite a complicated package structure where we want to have packages defined similar to the below:
Case 1
Package common1_x 0.1.0
requirements:
build:
- python
- numpy >=1.9
- pandas >=0.15.2
- setuptools
run:
- python
- numpy >=1.9
- pandas >=0.15.2
- nose
- coverage
Package common2_x 0.2.0
requirements:
build:
- python
- numpy >=1.9
- pandas >=0.15.2
- setuptools
run:
- python
- numpy >=1.9
- pandas >=0.15.2
- common1_x
Package toplevel_x 1.0.0
requirements:
build:
- python
run:
- python 2.7*
- numpy 1.9.2 py27_0
- pandas 0.15.2 np19py27_1
- common1_x 0.1.0 py27_0
- common2_x 0.2.0 py27_0
If the above are built and put in a conda repository (example at https://github.com/rekcahpassyla/conda-issue-918-example/tree/master/case1/channel/win-64), the following error is observed (at the end of the stack trace)
$ conda create -p c:\condatest\envs\toplevel -c thechannel toplevel_x==1.0.0
Fetching package metadata: ..........
Solving package specifications: ..........An unexpected error has occurred, please consider sending the
following traceback to the conda GitHub issue tracker at:
https://github.com/conda/conda/issues
Include the output of the command 'conda info' in your report.
...
File "C:\dev\bin\Anaconda\lib\site-packages\conda\resolve.py", line 624, in gen_clauses
assert len(clause) > 1, '%s %r' % (fn1, ms)
AssertionError: pandas-0.15.2-np19py27_1.tar.bz2 MatchSpec(u'python-dateutil 2.4.1')
Case 2
Package common1_y 0.1.0
requirements:
build:
- python
- numpy >=1.9
- pandas >=0.15.2
- setuptools
run:
- python
- numpy 1.9.2 py27_0
- pandas 0.15.2 np19py27_1
- python-dateutil 2.4.1 py27_0
- pytz 2015.4 py27_0
- six 1.9.0 py27_0
- coverage 3.7.1 py27_0
- nose 1.3.7 py27_0
Package common2_y 0.2.0
requirements:
build:
- python
- numpy >=1.9
- pandas >=0.15.2
- setuptools
run:
- python
- common1_y 0.1.0 np19py27_0
Package toplevel_y 1.0.0
requirements:
build:
- python
run:
- python 2.7*
- numpy 1.9.2 py27_0
- pandas 0.15.2 np19py27_1
- common2_y 0.2.0 py27_0
Example channel files at https://github.com/rekcahpassyla/conda-issue-918-example/tree/master/case2/channel/win-64
$ conda create -p c:\condatest\envs\toplevel -c thechannel toplevel_y==1.0.0
Fetching package metadata: ..........
Solving package specifications: .........An unexpected error has occurred, please consider sending the
following traceback to the conda GitHub issue tracker at:
https://github.com/conda/conda/issues
...
File "C:\dev\bin\Anaconda\lib\site-packages\conda\resolve.py", line 624, in gen_clauses
assert len(clause) > 1, '%s %r' % (fn1, ms)
AssertionError: pandas-0.15.2-np19py27_1.tar.bz2 MatchSpec(u'python-dateutil 2.4.1')
According to the current behaviour, our toplevel package meta.yaml currently needs to look like the following (example channel at https://github.com/rekcahpassyla/conda-issue-918-example/tree/master/working/channel/win-64), with every dependency of every subpackage specified. It gets rapidly unworkable as we may have hierarchies 4 levels deep.
requirements:
build:
- python
run:
- python 2.7*
- numpy 1.9.2 py27_0
- pandas 0.15.2 np19py27_1
- common1 0.1.0 py27_0
- common2 0.2.0 py27_0
# pandas dependencies
- python-dateutil 2.4.1 py27_0
- pytz 2015.4 py27_0
- six 1.9.0 py27_0
# common1 dependencies
- coverage 3.7.1 py27_0
- nose 1.3.7 py27_0
Output of conda info:
$ conda info
Current conda install:
platform : win-64
conda version : 3.18.3
conda-build version : 1.13.0+451.g76ab604
python version : 2.7.10.final.0
requests version : 2.8.1
root environment : C:\dev\bin\Anaconda (writable)
default environment : C:\dev\bin\Anaconda
envs directories : C:\dev\bin\Anaconda\envs
package cache : C:\dev\bin\Anaconda\pkgs
channel URLs :
http://repo.continuum.io/pkgs/dev/win-64/
http://repo.continuum.io/pkgs/dev/noarch/
http://repo.continuum.io/pkgs/gpl/win-64/
http://repo.continuum.io/pkgs/gpl/noarch/
http://repo.continuum.io/pkgs/free/win-64/
http://repo.continuum.io/pkgs/free/noarch/
config file :
is foreign system : False
Now that #1702 is merged (and bug fixed!), eliminating at least some of the performance issues that motivated this hack, I agree it's time.
@rekcahpassyla, ignore my last message, sorry about that. I agree it's caused by this issue.
No worries- I've committed my example channels and packages at https://github.com/rekcahpassyla/conda-issue-918-example
I actually experimented with removing this check when building #1702, and I experienced no issues with it. I went ahead and pushed a hotfix PR for consideration. If the automated tests pass I see no reason not to merge it.
@rekcahpassyla, it's actually quite easy to remove that strictness check from resolve.py; check the PR I just attached to this issue. Is it possible for you to try manually editing your installation of conda to do a quick check?
Thanks, I can certainly try that. Will report back with results...
I did the following:
- Checked out #1766
python setup.py develop
conda info now says:
C:\dev\code\opensource\conda [(ea06cf0...)]> conda info
Current conda install:
platform : win-64
conda version : 3.18.3-12-gea06cf0
conda-build version : 1.13.0+451.g76ab604
python version : 2.7.10.final.0
requests version : 2.8.1
root environment : C:\dev\bin\Anaconda (writable)
default environment : C:\dev\bin\Anaconda
envs directories : C:\dev\bin\Anaconda\envs
package cache : C:\dev\bin\Anaconda\pkgs
channel URLs :
http://repo.continuum.io/pkgs/dev/win-64/
http://repo.continuum.io/pkgs/dev/noarch/
http://repo.continuum.io/pkgs/gpl/win-64/
http://repo.continuum.io/pkgs/gpl/noarch/
http://repo.continuum.io/pkgs/free/win-64/
http://repo.continuum.io/pkgs/free/noarch/
config file :
is foreign system : False
And, both previous problem cases now do install.
Case 1:
C:\dev\code\opensource\conda [(c9d93af...)]> conda create -p c:\condatest\envs\toplevel_case1_1702 -c thechannel toplevel_x==1.0.0
Fetching package metadata: ..........
Solving package specifications: ....................
Package plan for installation in environment c:\condatest\envs\toplevel_case1_1702:
The following packages will be downloaded:
package | build
---------------------------|-----------------
common1_x-0.1.0 | py27_0 1 KB
common2_x-0.2.0 | py27_0 1 KB
toplevel_x-1.0.0 | np19py27_0 1 KB
------------------------------------------------------------
Total: 3 KB
The following NEW packages will be INSTALLED:
common1_x: 0.1.0-py27_0
common2_x: 0.2.0-py27_0
coverage: 4.0-py27_0
msvc_runtime: 1.0.0-vc9_0 [vc9]
nose: 1.3.7-py27_0
numpy: 1.9.2-py27_0
pandas: 0.15.2-np19py27_1
pip: 7.1.2-py27_0
python: 2.7.10-3
python-dateutil: 2.4.1-py27_0
pytz: 2015.6-py27_0
setuptools: 18.4-py27_0
six: 1.10.0-py27_0
toplevel_x: 1.0.0-np19py27_0
wheel: 0.26.0-py27_1
Case 2
C:\dev\code\opensource\conda [(c9d93af...) +0 ~1 -0]> conda create -p c:\condatest\envs\toplevel_case2_1702 -c thechannel toplevel_y==1.0.0
Fetching package metadata: ..........
Solving package specifications: .................
Package plan for installation in environment c:\condatest\envs\toplevel_case2_1702:
The following packages will be downloaded:
package | build
---------------------------|-----------------
common2_y-0.2.0 | py27_0 1 KB
toplevel_y-1.0.0 | np19py27_0 1 KB
------------------------------------------------------------
Total: 2 KB
The following NEW packages will be INSTALLED:
common1_y: 0.1.0-np19py27_0
common2_y: 0.2.0-py27_0
coverage: 3.7.1-py27_0
msvc_runtime: 1.0.0-vc9_0 [vc9]
nose: 1.3.7-py27_0
numpy: 1.9.2-py27_0
pandas: 0.15.2-np19py27_1
pip: 7.1.2-py27_0
python: 2.7.10-3
python-dateutil: 2.4.1-py27_0
pytz: 2015.4-py27_0
setuptools: 18.4-py27_0
six: 1.9.0-py27_0
toplevel_y: 1.0.0-np19py27_0
wheel: 0.26.0-py27_1
Proceed ([y]/n)? n
Sorry for being dumb, I misread things and I should have checked out #1766 instead. Have tested again with the right thing checked out, and results updated to reflect this.
Great. #1766 is ready for prime time, IMO. And the hassles I had getting the test scripts fixed illustrate that indeed, the new pruning pass justifies removing the strictness < 3 test.
Many thanks!! This has been a big problem for us. We are much obliged.
Turns out that issue #1778 is a duplicate of this one. It looks bad, because it is caused by conda update anaconda. But in fact, it only occurs if 1) people have additional channels added to the search; 2) those channels have packages with identical names as anaconda packages; and 3) they have dependencies that anaconda does not.
So that one's going to be a rare bug, but still, it illustrates the need to have this issue fixed.
So that one's going to be a rare bug
I think it's not so rare as you might think. It is common for many libraries to have optional support for various features, depending on the dependencies available.
A couple examples:
- libtiff can be compiled with optional jpeg support (if you tell it where to find libjpeg).
- VTK-5 can optionally be built with support for Qt (if you tell it where to find Qt and PyQt).
As it stands right now, the Anaconda version of those packages don't include those features. Therefore, I have my own versions, but they are (1) on my own channel, (2) have the same package name as the Anaconda version and (3) have extra dependencies. Bingo!
Fair point. So let's get this one pushed out there.
Time to close this; it's no longer operative.
Hi there, thank you for your contribution to Conda!
This issue has been automatically locked since it has not had recent activity after it was closed.
Please open a new issue if needed.