Your tox / coverage setup should not work, but it does
rfleschenberg opened this issue · comments
tox installs the package into the virtualenv it creates, then runs the tests inside that virtualenv. However, the source option from .coveragerc points at the source tree, not at the installation inside the tox virtualenv. As a result, coverage should not collect any data, and you should get an empty coverage report. But you don't.
Apparently, the tests run at least partly not in the tox virtualenv, but in the source tree.
This can be confirmed when modifying the test command to show some debugging information (thanks to @nedbat for suggesting this):
commands =
coverage run --debug=sys,config,trace -a {envbindir}/py.test -s tests
Sample output:
tests/test_tx.py Tracing '/tmp/prometheus_async/prometheus_async/aio/__init__.py'
Right now, I have no idea what causes this. I will try to investigate this further. However, you probably want to fix this, since it may cause unintended effects further down the road. It may be worthwhile to investigate the changedir option of tox (thanks to @brechtm for the suggestion).
See also this tox issue: https://bitbucket.org/hpk42/tox/issues/54/chdir-away-from-toxini-directory-during
One possible workaround for the issue you should be seeing (but are not) is to use usedevelop=True in tox.ini (again, thanks @nedbat).
As an aside, py.test recommends not having an __init__.py in your tests dir. I don't think this is related, though.
I’m traveling so I can’t look into it deeper but it seems like a mistake most people do. ;) It probably works, because the source code within the tree and under tox are the same?
I think this is also the reason some people put their packages into a separate src directory, so they avoid accidental imports.
I know about the __init__.py in tests thing but sometimes I just like to import things in my tests… :)
For my project, I'm now making use of the changedir option, setting it to tox's {envtmpdir}. This directory is emptied before the test commands are executed, so no installed modules can be shadowed.
This had the annoying side-effect that the coverage report output contained the absolute path to my package's modules. To obtain relative paths, it is necessary to run coverage report from the site-packages directory where tox installs my package. See my tox.ini and coveralls.py (which should really be named coverage.py) how this can be done.
This approach does make it more difficult to combine --parallel runs though, I suppose. I'm currently only testing coverage in a single tox environment.
@brechtm have you looked at the [paths] option in the .coveragerc? http://coverage.readthedocs.org/en/coverage-4.0.1/config.html#paths
@nedbat Thanks. I was vaguely aware of the [paths] section :). That should indeed help combining the coverage output from several tox enviroments. Now if only we could coverage combine after all others when using detox...