Tests are failing
fabaff opened this issue · comments
The same issue was report as #4. But on Fedora. The RPM generation is failing as test_nose are not passing.
_____________________________________________________________________ test_nose _____________________________________________________________________
capsys = <_pytest.capture.CaptureFixture object at 0x7f3671e321d0>
def test_nose(capsys):
class ResultStoringMain(NoseMain):
def runTests(self):
self.testRunner = NoseRunner(stream=sys.stdout,
verbosity=self.config.verbosity,
config=self.config)
self.result = self.testRunner.run(self.test)
main = ResultStoringMain(
module=None,
argv=['x', join(example_dir, 'example_nose')]
)
> assert main.result.testsRun == 3
E assert 1 == 3
E + where 1 = <nose.result.TextTestResult run=1 errors=0 failures=0>.testsRun
E + where <nose.result.TextTestResult run=1 errors=0 failures=0> = <tests.test_doc_example.test_nose.<locals>.ResultStoringMain object at 0x7f3671e32a90>.result
tests/test_doc_example.py:50: AssertionError
_____________________________________________________________________ test_nose _____________________________________________________________________
capsys = <_pytest.capture.CaptureFixture object at 0x7f3671a8e240>
def test_nose(capsys):
class ResultStoringMain(NoseMain):
def runTests(self):
self.testRunner = NoseRunner(stream=sys.stdout,
verbosity=self.config.verbosity,
config=self.config)
self.result = self.testRunner.run(self.test)
main = ResultStoringMain(
module=None,
argv=['x', '-vs', join(functional_test_dir, 'nose')]
)
> assert main.result.testsRun == 9
E assert 2 == 9
E + where 2 = <nose.result.TextTestResult run=2 errors=0 failures=0>.testsRun
E + where <nose.result.TextTestResult run=2 errors=0 failures=0> = <tests.test_functional.test_nose.<locals>.ResultStoringMain object at 0x7f3671a8e630>.result
tests/test_functional.py:163: AssertionError
Same comment as on that issue:
Can you let me know the exact steps you use to set up the environment in which this happens?
You can see exactly how those 3 and 9 test counts are calculated by reading the code above.
Looking at the failures, they're only when using nose, so it would be good to know exactly what version of nose you have installed, along with other python packages, and indeed the version of python.
Can you let me know the exact steps you use to set up the environment in which this happens?
No special setup. Requirements installed with dnf then run rpmbuild. Happens also in a clean venv.
$ git clone https://github.com/cjw296/sybil.git
[fab@xyz repos]$ cd sybil/
[fab@xyz sybil]$ python3 -m venv .
[fab@xyz sybil]$ source bin/activate
(sybil) [fab@xyz sybil]$ python3 setup.py develop
(sybil) [fab@xyz sybil]$ pip3 install nose pytest pytest-cov
(sybil) [fab@xyz sybil]$ pytest-3 tests
================================================= test session starts =================================================
platform linux -- Python 3.7.3, pytest-3.9.3, py-1.7.0, pluggy-0.8.1 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/fab/Documents/repos/sybil, inifile: setup.cfg
plugins: cov-2.6.0
collected 42 items
tests/test_capture.py::test_basic PASSED [ 2%]
tests/test_capture.py::test_directive_indent_beyond_block PASSED [ 4%]
tests/test_capture.py::test_directive_indent_equal_to_block PASSED [ 7%]
tests/test_codeblock.py::test_basic PASSED [ 9%]
tests/test_codeblock.py::test_future_imports PASSED [ 11%]
tests/test_doc_example.py::test_pytest PASSED [ 14%]
tests/test_doc_example.py::test_unittest PASSED [ 16%]
tests/test_doc_example.py::test_nose FAILED [ 19%]
tests/test_doctest.py::test_pass PASSED [ 21%]
tests/test_doctest.py::test_fail PASSED [ 23%]
tests/test_doctest.py::test_fail_with_options PASSED [ 26%]
tests/test_doctest.py::test_literals PASSED [ 28%]
tests/test_doctest.py::test_min_indent PASSED [ 30%]
tests/test_doctest.py::test_tabs PASSED [ 33%]
tests/test_functional.py::test_pytest PASSED [ 35%]
tests/test_functional.py::test_unittest PASSED [ 38%]
tests/test_functional.py::test_nose FAILED [ 40%]
tests/test_skip.py::test_basic PASSED [ 42%]
tests/test_skip.py::test_conditional_edge_cases PASSED [ 45%]
tests/test_skip.py::test_conditional_full PASSED [ 47%]
tests/test_skip.py::test_bad PASSED [ 50%]
tests/test_sybil.py::TestRegion::test_repr PASSED [ 52%]
tests/test_sybil.py::TestExample::test_repr PASSED [ 54%]
tests/test_sybil.py::TestExample::test_evaluate_okay PASSED [ 57%]
tests/test_sybil.py::TestExample::test_evaluate_not_okay PASSED [ 59%]
tests/test_sybil.py::TestExample::test_evaluate_raises_exception PASSED [ 61%]
tests/test_sybil.py::TestDocument::test_add PASSED [ 64%]
tests/test_sybil.py::TestDocument::test_add_no_overlap PASSED [ 66%]
tests/test_sybil.py::TestDocument::test_add_out_of_order PASSED [ 69%]
tests/test_sybil.py::TestDocument::test_add_adjacent PASSED [ 71%]
tests/test_sybil.py::TestDocument::test_add_before_start PASSED [ 73%]
tests/test_sybil.py::TestDocument::test_add_after_end PASSED [ 76%]
tests/test_sybil.py::TestDocument::test_add_overlaps_with_previous PASSED [ 78%]
tests/test_sybil.py::TestDocument::test_add_overlaps_with_next PASSED [ 80%]
tests/test_sybil.py::TestDocument::test_example_path PASSED [ 83%]
tests/test_sybil.py::TestDocument::test_example_line_and_column PASSED [ 85%]
tests/test_sybil.py::TestSybil::test_parse PASSED [ 88%]
tests/test_sybil.py::TestSybil::test_all_paths PASSED [ 90%]
tests/test_sybil.py::TestSybil::test_all_paths_with_base_directory PASSED [ 92%]
tests/test_sybil.py::TestFiltering::test_excludes PASSED [ 95%]
tests/test_sybil.py::TestFiltering::test_filenames PASSED [ 97%]
tests/test_sybil.py::test_namespace PASSED [100%]
====================================================== FAILURES =======================================================
______________________________________________________ test_nose ______________________________________________________
capsys = <_pytest.capture.CaptureFixture object at 0x7f9c26e01c18>
def test_nose(capsys):
class ResultStoringMain(NoseMain):
def runTests(self):
self.testRunner = NoseRunner(stream=sys.stdout,
verbosity=self.config.verbosity,
config=self.config)
self.result = self.testRunner.run(self.test)
main = ResultStoringMain(
module=None,
argv=['x', join(example_dir, 'example_nose')]
)
> assert main.result.testsRun == 3
E assert 1 == 3
E + where 1 = <nose.result.TextTestResult run=1 errors=0 failures=0>.testsRun
E + where <nose.result.TextTestResult run=1 errors=0 failures=0> = <tests.test_doc_example.test_nose.<locals>.ResultStoringMain object at 0x7f9c26bad2e8>.result
tests/test_doc_example.py:50: AssertionError
______________________________________________________ test_nose ______________________________________________________
capsys = <_pytest.capture.CaptureFixture object at 0x7f9c267b2ac8>
def test_nose(capsys):
class ResultStoringMain(NoseMain):
def runTests(self):
self.testRunner = NoseRunner(stream=sys.stdout,
verbosity=self.config.verbosity,
config=self.config)
self.result = self.testRunner.run(self.test)
main = ResultStoringMain(
module=None,
argv=['x', '-vs', join(functional_test_dir, 'nose')]
)
> assert main.result.testsRun == 9
E assert 2 == 9
E + where 2 = <nose.result.TextTestResult run=2 errors=0 failures=0>.testsRun
E + where <nose.result.TextTestResult run=2 errors=0 failures=0> = <tests.test_functional.test_nose.<locals>.ResultStoringMain object at 0x7f9c267b2198>.result
tests/test_functional.py:163: AssertionError
------------------------------------------------ Captured stdout call -------------------------------------------------
nose.test_nose.load_tests ... ok
nose.test_other.test_it ... ok
----------------------------------------------------------------------
Ran 2 tests in 0.001s
OK
================================================== warnings summary ===================================================
tests/test_doc_example.py::test_pytest
/usr/lib/python3.7/site-packages/_pytest/compat.py:332: RemovedInPytest4Warning: usage of Session.Class is deprecated, please use pytest.Class instead
return getattr(object, name, default)
/usr/lib/python3.7/site-packages/_pytest/compat.py:332: RemovedInPytest4Warning: usage of Session.File is deprecated, please use pytest.File instead
return getattr(object, name, default)
/usr/lib/python3.7/site-packages/_pytest/compat.py:332: RemovedInPytest4Warning: usage of Session.Function is deprecated, please use pytest.Function instead
return getattr(object, name, default)
/usr/lib/python3.7/site-packages/_pytest/compat.py:332: RemovedInPytest4Warning: usage of Session.Instance is deprecated, please use pytest.Instance instead
return getattr(object, name, default)
/usr/lib/python3.7/site-packages/_pytest/compat.py:332: RemovedInPytest4Warning: usage of Session.Item is deprecated, please use pytest.Item instead
return getattr(object, name, default)
/usr/lib/python3.7/site-packages/_pytest/compat.py:332: RemovedInPytest4Warning: usage of Session.Module is deprecated, please use pytest.Module instead
return getattr(object, name, default)
tests/test_functional.py::test_pytest
/usr/lib/python3.7/site-packages/_pytest/compat.py:332: RemovedInPytest4Warning: usage of Session.Class is deprecated, please use pytest.Class instead
return getattr(object, name, default)
/usr/lib/python3.7/site-packages/_pytest/compat.py:332: RemovedInPytest4Warning: usage of Session.File is deprecated, please use pytest.File instead
return getattr(object, name, default)
/usr/lib/python3.7/site-packages/_pytest/compat.py:332: RemovedInPytest4Warning: usage of Session.Function is deprecated, please use pytest.Function instead
return getattr(object, name, default)
/usr/lib/python3.7/site-packages/_pytest/compat.py:332: RemovedInPytest4Warning: usage of Session.Instance is deprecated, please use pytest.Instance instead
return getattr(object, name, default)
/usr/lib/python3.7/site-packages/_pytest/compat.py:332: RemovedInPytest4Warning: usage of Session.Item is deprecated, please use pytest.Item instead
return getattr(object, name, default)
/usr/lib/python3.7/site-packages/_pytest/compat.py:332: RemovedInPytest4Warning: usage of Session.Module is deprecated, please use pytest.Module instead
return getattr(object, name, default)
-- Docs: https://docs.pytest.org/en/latest/warnings.html
================================== 2 failed, 40 passed, 12 warnings in 0.67 seconds ===================================
(sybil) [fab@xyz sybil]$ Looking at the failures, they're only when using nose, so it would be good to know exactly what version of nose you have installed, along with other python packages, and indeed the version of python.
$ rpm -q python3
python3-3.7.3-3.fc30.x86_64$ rpm -q python3-nose
python3-nose-1.3.7-22.fc30.noarchIn the venv:
(sybil) [fab@xyz sybil]$ pip3 freeze
atomicwrites==1.3.0
attrs==19.1.0
coverage==4.5.3
importlib-metadata==0.18
more-itertools==7.0.0
nose==1.3.7
packaging==19.0
pluggy==0.12.0
py==1.8.0
pyparsing==2.4.0
pytest==4.6.3
pytest-cov==2.7.1
six==1.12.0
-e git+https://github.com/cjw296/sybil.git@3062d531b672a7943f91f030dc160923c833dff8#egg=sybil
wcwidth==0.1.7
zipp==0.5.1Something is amiss in your setup. Note that your pip3 freeze and pytest output show different versions of pytest, nb:
(sybil) [fab@xyz sybil]$ pytest-3 tests
================================================= test session starts =================================================
platform linux -- Python 3.7.3, pytest-3.9.3, py-1.7.0, pluggy-0.8.1 -- /usr/bin/python3
...versus:
(sybil) [fab@xyz sybil]$ pip3 freeze
...
pytest==4.6.3
sybil's setup.py does only say in needs 3.5+, but that doesn't explain what you're seeing:
https://github.com/cjw296/sybil/blob/master/setup.py#L35
I wonder if 6461d81 actually mean's sybil's tests now rely on pytest 4.0+?
Anyway, if you can figure out what's going on with your env above, we may have an answer...
Sorry, my bad. I invoked the system-wide pytest instance which is 3.9.3 and not 4.6.3 which was installed in the venv.
It looks like that 3.x of pytest is not working for the nose part. The should be 4+.
Yeah, this is a difficult one; I don't want to change the version in setup.py because I think sybil actually works fine with pytest 3.9, however sybil's tests make assertions that are highly dependent on the version of pytest used, and that's somewhat unavoidable.
I'll close this as resolved for now.
No, it doesn't work with pytest-3.9.3 on a Fedora installation and it will most likely not work on others like Debian which is mentioned in #4. If the setup.py says that 'pytest>=3.5.0' is the minimal requirement then the tests must pass with 3.5.0.
The RPM build process is failing because of this (Fedora 30 ships 3.9.3 which is >=3.5.0 and only Fedora Rawhide has pytest 4+). Thus the requirements are fulfilled for F30. But now we know that the tests only pass with pytest 4+.
The required release of pytest for sybil has to be adjusted. Please consider to change it.
Otherwise every package maintainer has to investigate the issue and probably patch it. The runtime requirements for Fedora are already automatically collected. I assume that the same will happen for the testing requirements and then it would require a patch for sure.
"it doesn't work with pytest-3.9.3 on a Fedora installation" - what evidence do you have of this?
(the failing tests aren't sufficient in this case, I'm afraid...)
"it doesn't work with pytest-3.9.3 on a Fedora installation" - what evidence do you have of this?
Like the output from the build process...
================================================= test session starts =================================================
platform linux -- Python 3.7.3, pytest-3.9.3, py-1.7.0, pluggy-0.8.1 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/fab/Documents/repos/rpmbuild/BUILD/sybil-1.2.0, inifile: setup.cfg
plugins: cov-2.6.0, aspectlib-1.4.2
[...]All requirements are fulfilled but this doesn't make the test pass. In a venv you will have the latest releases but distribution packages are not always up-to-date.
(the failing tests aren't sufficient in this case, I'm afraid...)
Well, they are. Failing tests are blocking the generation of RPM/DEB packages when the tests are activated and this is preferred. In your own comment you suggest to use pytest 4+.
Okay, so tell me what the minimum pytest version is that works for you?
Actually these problems are caused by the nose plugin entry point being ignored, for example because the module is tested without being installed. Setting PYTHONPATH or running pytest as python -m pytest may be needed to fix this, not sure if this is actually dependent on the pytest version.
Right. Honestly, the only sane way to run sybil's tests are to test it like its own CI does...
I can reproduce this issue on Alpine Linux. Same setup as Fedora really, dependencies installed via system package manager, tests are invoked with PYTHONPATH="$PWD/build/lib" pytest to make sure they can be run without having the package installed yet.
However, PYTHONPATH="$PWD/build/lib" python3 -m pytest actually makes it pass successfully.
Just noting this for reference 😉
Manually hacking around with PYTHONPATH isn't sane.
Trying to test a package without installing it isn't sane.
I'm sorry system packagers feel such a need to make things harder for themselves.