Use pytest to run all tests
macfreek opened this issue · comments
ConfigArgParse uses setuptools (using python setup.py test
) to run all tests. However, this gives a WARNING: Testing via this command is deprecated and will be removed in a future version.
. pytest is said to be an drop-in replacement for this.
Running pytest reveals a few minor issues (1665 of the 1666 tests work fine). I'll enumerate them here.
testConfigOrEnvValueErrors
fails
(tests.test_configargparse.TestBasicUseCases.testConfigOrEnvValueErrors
)
testConfigOrEnvValueErrors SUCCEEDS for python setup.py test
testConfigOrEnvValueErrors SUCCEEDS for pytest
or pytest -k testConfigOrEnvValueErrors
testConfigOrEnvValueErrors FAILS for pytest -v
or pytest -v -k testConfigOrEnvValueErrors
The -v
parameter means that pytest
is verbose. testConfigOrEnvValueErrors tests the VERBOSE environment variable (amongst other things). I suspect that this test incorrectly fails due to the way pytest
sets or influences by the environment variables.
TestHelpFormattingMetaclass
gives a warning
(tests.test_argparse.TestHelpFormattingMetaclass
)
Pytests gives a PytestCollectionWarning: cannot collect test class 'TestHelpFormattingMetaclass' because it has a __init__ constructor (from: tests/test_configargparse.py)
.
This is a bit of a pickle. TestHelpFormattingMetaclass
is defined in Python's test_argparse.py, and we can't change the code.
This class contains a few tests that we perhaps like to run, but neither unittest nor pytest collects any of these tests because these tests are dynamically added to the class, and seemingly neither unittest nor pytest can handle that situation. The only difference is that pytest gives a warning about it.
I tried to suppress the warning by setting
test.test_argparse.TestHelpFormattingMetaclass.__test__ = False
But so far, I still see the warning.
testGlobalInstances
gives confusing output
(tests.test_configargparse.TestMisc.testGlobalInstances
and tests.test_configargparse.TestMisc.testGlobalInstances_WithName
)
These tests both contain code that tests if a ValueError is raised under certain conditions (in the self.assertRaisesRegex
line). This seems to work fine, and the tests seem to pass. At least, they always pass according to the summary line and the lines
tests/test_configargparse.py::TestMisc::testGlobalInstances PASSED
tests/test_configargparse.py::TestMisc::testGlobalInstances_WithName PASSED
If only theses two tests are run with pytest -k testGlobalInstances -v
, there are no errors, even with the verbose (-v
) option set. However, if all tests are run with pytest -v
, it presents us with two errors in the stdout:
testGlobalInstances (tests.test_configargparse.TestMisc) ... ERROR
testGlobalInstances_WithName (tests.test_configargparse.TestMisc) ... ERROR
In addition, the two ValueErrror exceptions are shows in the test summary, even though the tests passes:
tests/test_configargparse.py::TestMisc::testGlobalInstances PASSED
tests/test_configargparse.py::TestMisc::testGlobalInstances_WithName PASSED
It's unclear to me why these exceptions are shown so specifically. I find it confusing, and can't seem to find a way to suppress these warnings.
ResourceWarning: unclosed file
When running python setup.py test
, mock
generate a ResourceWarning: unclosed file
for three tests. The test itself succeeds.
testBasicCase2 (tests.test_configargparse.TestBasicUseCases) ...
/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/traceback.py:220: ResourceWarning: unclosed file <_io.TextIOWrapper name='/var/folders/zx/y85xrx5x0lz6bgqd45lhr9kc0000gp/T/tmpqe8s6ado' mode='r' encoding='UTF-8'>
tb.tb_frame.clear()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
testBasicCase2_WithGroups (tests.test_configargparse.TestBasicUseCases) ...
/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/traceback.py:220: ResourceWarning: unclosed file <_io.TextIOWrapper name='/var/folders/zx/y85xrx5x0lz6bgqd45lhr9kc0000gp/T/tmpwocvu164' mode='r' encoding='UTF-8'>
tb.tb_frame.clear()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
testKwrgsArePassedToArgParse (tests.test_configargparse.TestMisc) ...
/Users/freek/Library/Python/3.8/lib/python/site-packages/mock/mock.py:2072: ResourceWarning: unclosed file <_io.FileIO name=3 mode='rb+' closefd=True>
setattr(_type, entry, MagicProxy(entry, self))
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/Users/freek/Library/Python/3.8/lib/python/site-packages/mock/mock.py:2072: ResourceWarning: unclosed file <_io.FileIO name=4 mode='rb+' closefd=True>
setattr(_type, entry, MagicProxy(entry, self))
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/Users/freek/Library/Python/3.8/lib/python/site-packages/mock/mock.py:2072: ResourceWarning: unclosed file <_io.FileIO name=5 mode='rb+' closefd=True>
setattr(_type, entry, MagicProxy(entry, self))
ResourceWarning: Enable tracemalloc to get the object allocation traceback
The failure of testConfigOrEnvValueErrors
is because args
is not set when calling TestCase.assertParseArgsRaises
. This causes args
to be set to sys.args
. So calling this test with pytest -k testConfigOrEnvValueErrors -v
would give a parser error "unrecognized arguments: -k testConfigOrEnvValueErrors
". That clearly is not intended: these tests should not depend on how the test suite was called. I'll make a PR.
Still looking into the other issues, since I still get some unexpected result.
I finally understand what happened. The argparse
module contains a function called test_main()
which runs all tests. pytest thinks this is a test function itself and calls it as such, which runs all tests again, and also produced the weird stdout of testGlobalInstances that I've seen.
Thanks to the way configargparse is importing tests_argparse, it is possible to rename both TestHelpFormattingMetaclass
and test_main()
to prevent this behaviour.
The ResourceWarning: unclosed file
for testBasicCase2
and testBasicCase2_WithGroups
are caused by a minor issue in ArgumentParser._open_config_files()
.
ArgumentParser._open_config_files()
opens multiple configuration files, and returns a list of the file streams. If one of the file open()
operations fails, an exception is raised. E.g. file not found.
However, this never returns nor closes the previously opened files, leaving dangling file pointers. This commit fixes the issue by closing previously opened config files when such an exception occurs.
And the ResourceWarning: unclosed file
in testKwrgsArePassedToArgParse
and testGlobalInstances_WithName
did not originate there, but in the test(s) that were run just before that: testConstructor_WriteOutConfigFileArgs
and friends.
In testConstructor_WriteOutConfigFileArgs
, testConstructor_WriteOutConfigFileArgs2
and testConstructor_WriteOutConfigFileArgsLong
, a temporary file cfg_f
is created that was never closed.
And with that, I think I caught all errors and warnings in the test suite, so I can make a PR and start adding my own tests (knowing if any issues occur, it is because of my code and not existing code). I'll try to do that later today.
All above issues are addressed in PR #214.
Since #214 is merged, this can be closed.