Not displaying modified/new file coverage, only repo-wide
Adrian-at-CrimsonAzure opened this issue · comments
Title says it all, we're only getting the repo-wide total:
This PR adds two new files and their tests:
I'd be happy to provide sanitized logs or configs if needed.
I think this is happening because tests are done on package which is installed via pip, so coverage report has something like:
---------- coverage: platform linux, python 3.8.16-final-0 -----------
Name Stmts Miss Cover Missing
------------------------------------------------------------------------------------------------------------------------------------
/tmp/tmp.PCzKULX5yI/venv/lib/python3.8/site-packages/random_forestry/__init__.py 2 0 100%
/tmp/tmp.PCzKULX5yI/venv/lib/python3.8/site-packages/random_forestry/forestry.py 336 71 79% 22, 323-325, 335-337, 340-344, 347-351, 354-365, 370, 384, 587, 598, 616, 632, 634, 636, 647, 669, 687, 689, 691, 702, 725, 732-767, 921-924, 977-994, 1006-1019, 1067, 1204-1206, 1256-1259, 1272-1273
/tmp/tmp.PCzKULX5yI/venv/lib/python3.8/site-packages/random_forestry/preprocessing.py 105 29 72% 14-22, 38, 48-66, 83, 88, 141, 153, 203, 220-227, 249
/tmp/tmp.PCzKULX5yI/venv/lib/python3.8/site-packages/random_forestry/processed_dta.py 25 0 100%
/tmp/tmp.PCzKULX5yI/venv/lib/python3.8/site-packages/random_forestry/test.py 30 30 0% 1-50
/tmp/tmp.PCzKULX5yI/venv/lib/python3.8/site-packages/random_forestry/validators/__init__.py 8 0 100%
/tmp/tmp.PCzKULX5yI/venv/lib/python3.8/site-packages/random_forestry/validators/base_validator.py 11 1 91% 15
/tmp/tmp.PCzKULX5yI/venv/lib/python3.8/site-packages/random_forestry/validators/fit_validator.py 222 29 87% 46, 57, 82, 85, 88, 91, 101, 104, 107, 110, 118, 134, 137, 144, 156, 159, 164, 171, 214, 222, 237, 256, 273, 281, 284, 287, 301, 304, 310
/tmp/tmp.PCzKULX5yI/venv/lib/python3.8/site-packages/random_forestry/validators/predict_validator.py 75 8 89% 21, 36, 55, 69, 95-98
------------------------------------------------------------------------------------------------------------------------------------
TOTAL 814 168 79%
These filepaths doesn't match with what git reports as new/modified files.
@orgoro what do you think about adding support for such use case?
how are you running your coverage? can you include an example output?
Here is my full example.
My repo/branch is Rforestry, and gh action results are here
Explanation:
Folder Python in repository contains python package we build and test, and Python/random_forestry contains source code. The package also contains C++ extension, so we have to build everything first, before testing. pip install does it all for us, and it also puts package in system wide site-packages, and when we run pytest, it's using the package from there, not from cloned repo. Because of that, we set --cov=random_forestry as pytest argument, and in this case, random_forestry is package name, not folder name (obviously, we can't run pytest from folder Python because it contains folder random_forestry so it will get confused and report 0% coverage). The output looks like in my comment above. Since we are using package name in --cov and not source folder, resulting coverage.xml has string <source></source>, so variable source in your gh action source code evaluates to empty string and totally mess up results.
I forked your repo here, and made small change on src/coverage.ts [lines 46-47], and it works for my case, but it requires some extra work to make it general.
I'm having the same issue, though in my case I think it's because I run my tests within a container. In the output from the action it shows:
git new files: [] modified files: ["bin/test", "name-of-app/main.py"]
source: /app/name-of-app
Which I think means that it'll only consider things that have a prefix of '/app/name-of-app' as their filename in git, which nothing will have as that's the path within the container. (With 'name-of-app' being the top level directory I'd want reported on as that's where all the python files are based.
I think to cover all the cases, maybe add some more variables we can set on the action, such as allowing us to specify a (regex?) prefix to remove from the reported paths to get a match to the paths in git? (Which I'd set to '/app/' and @petrovicboban would set to '/tmp/tmp.*/venv/lib/python3.8/site-packages/' ?
gh action output in my case is:
git new files: [] modified files: [".github/workflows/python.yaml",".gitignore","Python/pyproject.toml","Python/random_forestry/forestry.py","Python/tests/test_constructor.py","Python/tests/test_forestry.py","Python/tests/test_forests_equality.py"]
source:
and filepaths in coverage.xml are like in comment above:
---------- coverage: platform linux, python 3.8.16-final-0 -----------
Name Stmts Miss Cover Missing
------------------------------------------------------------------------------------------------------------------------------------
/tmp/tmp.PCzKULX5yI/venv/lib/python3.8/site-packages/random_forestry/__init__.py 2 0 100%
/tmp/tmp.PCzKULX5yI/venv/lib/python3.8/site-packages/random_forestry/forestry.py 336 71 79% 22, 323-325, 335-337, 340-344, 347-351, 354-365, 370, 384, 587, 598, 616, 632, 634, 636, 647, 669, 687, 689, 691, 702, 725, 732-767, 921-924, 977-994, 1006-1019, 1067, 1204-1206, 1256-1259, 1272-1273
/tmp/tmp.PCzKULX5yI/venv/lib/python3.8/site-packages/random_forestry/preprocessing.py 105 29 72% 14-22, 38, 48-66, 83, 88, 141, 153, 203, 220-227, 249
/tmp/tmp.PCzKULX5yI/venv/lib/python3.8/site-packages/random_forestry/processed_dta.py 25 0 100%
/tmp/tmp.PCzKULX5yI/venv/lib/python3.8/site-packages/random_forestry/test.py 30 30 0% 1-50
/tmp/tmp.PCzKULX5yI/venv/lib/python3.8/site-packages/random_forestry/validators/__init__.py 8 0 100%
/tmp/tmp.PCzKULX5yI/venv/lib/python3.8/site-packages/random_forestry/validators/base_validator.py 11 1 91% 15
/tmp/tmp.PCzKULX5yI/venv/lib/python3.8/site-packages/random_forestry/validators/fit_validator.py 222 29 87% 46, 57, 82, 85, 88, 91, 101, 104, 107, 110, 118, 134, 137, 144, 156, 159, 164, 171, 214, 222, 237, 256, 273, 281, 284, 287, 301, 304, 310
/tmp/tmp.PCzKULX5yI/venv/lib/python3.8/site-packages/random_forestry/validators/predict_validator.py 75 8 89% 21, 36, 55, 69, 95-98
------------------------------------------------------------------------------------------------------------------------------------
TOTAL
so what I did in src/coverage.ts was to remove prefix Python/ from files reported by git:
46: const fileName = file.replace(/Python\//, '').replace(/\//g, '\\/')
and then do regex on filepaths reported by coverage:
47: const regex = new RegExp(`.*filename=".*/${fileName}".*line-rate="(?<cover>[0-9]+[.]*[0-9]*)".*`)
Note .*/ in front of ${fileName} which covers /tmp/tmp.PCzKULX5yI/venv/lib/python3.8/site-packages/ in my case.
Other than adding regex to inputs, as proposed by @dragonpaw , new input variable for deleting prefix for git reported paths can be of use, because in some cases such as mine, source is empty string, and python packages can be nested anywhere in repo.
@dragonpaw what gh action does is stripping source from files reported by git, and then look for such filename in coverage.xml. Investigate your coverage.xml, or just add extra output as mine (--cov-report=term), so you don't have to look at coverage.xml. Then try to experiement, with changing working directory when you run pytest, and source in --cov=<source>
For me the coverage report looks like:
Name Stmts Miss Cover
-------------------------------------------------------------------------------------------------------------------------
name-of-app/__init__.py 3 0 100%
...So the report matches the git paths exactly, but I'm guessing the 'source' getting read as '/app/name-of-app' is what's throwing it off that it thinks no changed files match when on several PRs we've had multiple changed files and they don't show up in the list of changed files that this action posts.
@dragonpaw can you try with altering relative_files in coverage config?
https://coverage.readthedocs.io/en/6.2/config.html#run
@dragonpaw can you try with altering
relative_filesincoverageconfig? https://coverage.readthedocs.io/en/6.2/config.html#run
Setting relative=true (in pyproject.toml in my case) did indeed fix it so that it now shows the per-file coverage stats.
Hi there,
I'm having the same problem. Tried the @dragonpaw solutions (and many others) but no good news for anybody.
Here's my repo:
https://github.com/llGines/checkCalcCoverage
Is a quite simple project, just for testing yours.
Many thanks.
Hi there,
I'm having the same problem. Tried the @dragonpaw solutions (and many others) but no good news for anybody. Here's my repo: https://github.com/llGines/checkCalcCoverage
Is a quite simple project, just for testing yours. Many thanks.
what worked for us:
Using the environment variable {Python_ROOT_DIR?} provided by actions/setup-python@v4
and setting source= ${Python_ROOT_DIR?}/**/site-packages/