Fine-tune coverage execution
Zearin opened this issue · comments
Problem
Part I
The documentation for coverage states that GEvent can confuse coverage’s results:
gevent, which is based on greenlet, and is similar to eventlet. All of these manipulate the C stack, and therefore confuse coverage.py. Issue 149 has some pointers to more information.
Part II
Should we be using --timid?
If your coverage results seem to be overlooking code that you know has been executed, try running coverage again with the
--timidflag. This uses a simpler but slower trace method.
Source: http://nedbatchelder.com/code/coverage/cmd.html#cmd-execution
Part III
Should we be using --parallel-mode?
If you are measuring coverage in a multi-process program, or across a number of machines, you’ll want the
--parallel-modeswitch to keep the data separate during measurement. See Combining data files below.
Source: http://nedbatchelder.com/code/coverage/cmd.html#cmd-execution
Summary
- Investigate coverage + GEvent
- Investigate
--timid - Investigate
--parallel-mode
Have you tried any of those?
It seems to me that it would be useful to have tests around these issues before fixing them. Can you write the tests that prove the issue?
Thanks!!!
Makes sense. ☺ Will try to test this shortly.
coverage + GEvent
Confirmed.
On coverage’s gevent issue, Ned Batchelder said:
Here's a fix, though it sacrifices other behavior: https://github.com/newbrough/coverage
Here is coverage as pyvows currently stands:
=============
Code Coverage
=============
✗ pyvows/__init__ 0.0% 2, 36, 37, and 4 more
✗ pyvows/__main__ 0.0% 3, 5, 7, and 2 more
✗ pyvows/assertions/__init__ 0.0% 3, 25, 26, and 11 more
✗ pyvows/color 0.0% 2, 16, 17, and 35 more
✗ pyvows/reporting/__init__ 0.0% 11, 12, 13, and 4 more
✗ pyvows/reporting/profile 0.0% 2, 11, 13, and 31 more
✗ pyvows/version 0.0% 2, 13, 16, and 1 more
✗ pyvows/reporting/common •• 4.3% 2, 12, 14, and 63 more
✗ pyvows/cli •• 5.5% 3, 14, 16, and 101 more
✗ pyvows/reporting/coverage •••••• 13.2% 2, 12, 14, and 63 more
✗ pyvows/assertions/types/numeric •••••••••• 20.0% 3, 14, 16, and 1 more
✗ pyvows/assertions/emptiness •••••••••••• 25.0% 3, 14, 17
✗ pyvows/assertions/equality •••••••••••• 25.0% 3, 14, 17
✗ pyvows/assertions/inclusion •••••••••••• 25.0% 3, 14, 17
✗ pyvows/reporting/test ••••••••••••••• 30.6% 2, 11, 13, and 47 more
✗ pyvows/errors ••••••••••••••••• 35.0% 2, 13, 18, and 10 more
✗ pyvows/result •••••••••••••••••••• 41.5% 2, 15, 23, and 21 more
✗ pyvows/assertions/types/function •••••••••••••••••••••• 44.4% 3, 14, 16, and 2 more
✗ pyvows/assertions/types/regexp •••••••••••••••••••••• 44.4% 3, 14, 16, and 2 more
✗ pyvows/assertions/types/errors •••••••••••••••••••••• 45.5% 3, 14, 16, and 3 more
✗ pyvows/async_topic •••••••••••••••••••••••• 48.5% 2, 15, 18, and 14 more
✗ pyvows/reporting/xunit •••••••••••••••••••••••• 49.1% 2, 12, 13, and 26 more
✗ pyvows/assertions/length ••••••••••••••••••••••••• 50.0% 3, 14, 17, and 1 more
✗ pyvows/assertions/types/boolean ••••••••••••••••••••••••• 50.0% 3, 14, 17, and 1 more
✗ pyvows/assertions/types/classes ••••••••••••••••••••••••• 50.0% 3, 14, 17, and 1 more
✗ pyvows/assertions/types/nullable ••••••••••••••••••••••••• 50.0% 3, 14, 17, and 1 more
✗ pyvows/core ••••••••••••••••••••••••• 51.1% 2, 12, 13, and 62 more
✗ pyvows/assertions/types/file •••••••••••••••••••••••••••••••• 65.0% 3, 18, 19, and 4 more
✗ pyvows/assertions/like •••••••••••••••••••••••••••••••••• 68.6% 3, 19, 21, and 13 more
✗ pyvows/runner ••••••••••••••••••••••••••••••••••• 70.6% 2, 15, 16, and 50 more
✗ OVERALL •••••••••••••••••• 36.7%
============
Vows Results
============
✓ OK » 497 honored • 0 broken (0.959065s)
Next, I uninstalled coverage, then re-installed it directly from newbrough/coverage.
The difference is not huge, but it’s there:
=============
Code Coverage
=============
✗ pyvows.__init__ 0.0% 2, 36, 37, and 4 more
✗ pyvows.__main__ 0.0% 3, 5, 7, and 2 more
✗ pyvows.color 0.0% 2, 16, 17, and 35 more
✗ pyvows.version 0.0% 2, 13, 16, and 1 more
✗ pyvows.assertions.__init__ 0.0% 3, 25, 26, and 11 more
✗ pyvows.reporting.__init__ 0.0% 11, 12, 13, and 4 more
✗ pyvows.reporting.profile 0.0% 2, 11, 13, and 31 more
✗ pyvows.reporting.common •• 4.3% 2, 12, 14, and 63 more
✗ pyvows.cli ••• 7.3% 3, 14, 16, and 99 more
✗ pyvows.reporting.coverage •••••• 13.2% 2, 12, 14, and 63 more
✗ pyvows.assertions.types.numeric •••••••••• 20.0% 3, 14, 16, and 1 more
✗ pyvows.assertions.emptiness •••••••••••• 25.0% 3, 14, 17
✗ pyvows.assertions.equality •••••••••••• 25.0% 3, 14, 17
✗ pyvows.assertions.inclusion •••••••••••• 25.0% 3, 14, 17
✗ pyvows.reporting.test ••••••••••••••• 30.6% 2, 11, 13, and 47 more
✗ pyvows.errors ••••••••••••••••• 35.0% 2, 13, 18, and 10 more
✗ pyvows.result •••••••••••••••••••• 41.5% 2, 15, 23, and 21 more
✗ pyvows.assertions.types.function •••••••••••••••••••••• 44.4% 3, 14, 16, and 2 more
✗ pyvows.assertions.types.regexp •••••••••••••••••••••• 44.4% 3, 14, 16, and 2 more
✗ pyvows.assertions.types.errors •••••••••••••••••••••• 45.5% 3, 14, 16, and 3 more
✗ pyvows.async_topic •••••••••••••••••••••••• 48.5% 2, 15, 18, and 14 more
✗ pyvows.reporting.xunit •••••••••••••••••••••••• 49.1% 2, 12, 13, and 26 more
✗ pyvows.assertions.length ••••••••••••••••••••••••• 50.0% 3, 14, 17, and 1 more
✗ pyvows.assertions.types.boolean ••••••••••••••••••••••••• 50.0% 3, 14, 17, and 1 more
✗ pyvows.assertions.types.classes ••••••••••••••••••••••••• 50.0% 3, 14, 17, and 1 more
✗ pyvows.assertions.types.nullable ••••••••••••••••••••••••• 50.0% 3, 14, 17, and 1 more
✗ pyvows.core ••••••••••••••••••••••••• 51.1% 2, 12, 13, and 62 more
✗ pyvows.assertions.types.file •••••••••••••••••••••••••••••••• 65.0% 3, 18, 19, and 4 more
✗ pyvows.assertions.like •••••••••••••••••••••••••••••••••• 68.6% 3, 19, 21, and 13 more
✗ pyvows.runner •••••••••••••••••••••••••••••••••••• 72.2% 2, 15, 16, and 47 more
✗ OVERALL •••••••••••••••••• 37.2%
============
Vows Results
============
✓ OK » 497 honored • 0 broken (0.927781s)