Wrong coverage when there's a pytest plugin installed that imports the module that you want to cover

[Nelson Yeung]

Summary

When there's a pytest plugin installed that imports the module that you want to have coverage, it will not cover that module. However, running coverage run -m pytest ... works as expected.

Expected vs actual result

Expected:

src/foo/__init__.py       2      0   100%

Actual result:

src/foo/__init__.py       2      1    50%

Reproducer

Create src/foo/__init__.py:

def foo() -> str:
    return "foo"

Create src/pytest_foo/__init__.py:

import foo

Create tests/test_foo.py:

import foo
def test_foo():
    assert foo.foo() == "foo"

I'm using poetry for the build, so in my pyproject.toml I have:

[tool.poetry]
name = "foo"
version = "1.0.0"
description = ""
authors = ["foo bar"]
packages = [
  {include = "foo", from = "src"},
  {include = "pytest_foo", from = "src"},
]

[tool.poetry.plugins.pytest11]
foo = "pytest_foo"

[tool.pytest.ini_options]
addopts = [
  "--cov-fail-under=80",
  "--cov=foo",
]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Run pytest:

tests/test_foo.py .                                                                                                                                                                                                                                                      [100%]

---------- coverage: platform darwin, python 3.7.13-final-0 ----------
Name                  Stmts   Miss  Cover
-----------------------------------------
src/foo/__init__.py       2      1    50%
-----------------------------------------
TOTAL                     2      1    50%

FAIL Required test coverage of 80% not reached. Total coverage: 50.00%

Versions

pytest==7.2.2
pytest-cov==4.0.0

[Nelson Yeung]

I realised that although this is a slightly different problem than https://pytest-cov.readthedocs.io/en/latest/plugins.html (since I'm not even trying to cover the plugin itself just the API it uses separately), it's probably caused by the same underlying problem with the plugin system. As a workaround, I wrote a script that generates a .pth file and put it into the site-packages directory which sets the coverage variables when running pytest.