Use pytest to run all tests
macfreek opened this issue · comments
ConfigArgParse uses setuptools (using python setup.py test) to run all tests. However, this gives a WARNING: Testing via this command is deprecated and will be removed in a future version.. pytest is said to be an drop-in replacement for this.
Running pytest reveals a few minor issues (1665 of the 1666 tests work fine). I'll enumerate them here.
testConfigOrEnvValueErrors fails
(tests.test_configargparse.TestBasicUseCases.testConfigOrEnvValueErrors)
testConfigOrEnvValueErrors SUCCEEDS for python setup.py test
testConfigOrEnvValueErrors SUCCEEDS for pytest or pytest -k testConfigOrEnvValueErrors
testConfigOrEnvValueErrors FAILS for pytest -v or pytest -v -k testConfigOrEnvValueErrors
The -v parameter means that pytest is verbose. testConfigOrEnvValueErrors tests the VERBOSE environment variable (amongst other things). I suspect that this test incorrectly fails due to the way pytest sets or influences by the environment variables.
TestHelpFormattingMetaclass gives a warning
(tests.test_argparse.TestHelpFormattingMetaclass)
Pytests gives a PytestCollectionWarning: cannot collect test class 'TestHelpFormattingMetaclass' because it has a __init__ constructor (from: tests/test_configargparse.py).
This is a bit of a pickle. TestHelpFormattingMetaclass is defined in Python's test_argparse.py, and we can't change the code.
This class contains a few tests that we perhaps like to run, but neither unittest nor pytest collects any of these tests because these tests are dynamically added to the class, and seemingly neither unittest nor pytest can handle that situation. The only difference is that pytest gives a warning about it.
I tried to suppress the warning by setting
test.test_argparse.TestHelpFormattingMetaclass.__test__ = False
But so far, I still see the warning.
testGlobalInstances gives confusing output
(tests.test_configargparse.TestMisc.testGlobalInstances and tests.test_configargparse.TestMisc.testGlobalInstances_WithName)
These tests both contain code that tests if a ValueError is raised under certain conditions (in the self.assertRaisesRegex line). This seems to work fine, and the tests seem to pass. At least, they always pass according to the summary line and the lines
tests/test_configargparse.py::TestMisc::testGlobalInstances PASSED
tests/test_configargparse.py::TestMisc::testGlobalInstances_WithName PASSED
If only theses two tests are run with pytest -k testGlobalInstances -v, there are no errors, even with the verbose (-v) option set. However, if all tests are run with pytest -v, it presents us with two errors in the stdout:
testGlobalInstances (tests.test_configargparse.TestMisc) ... ERROR
testGlobalInstances_WithName (tests.test_configargparse.TestMisc) ... ERROR
In addition, the two ValueErrror exceptions are shows in the test summary, even though the tests passes:
tests/test_configargparse.py::TestMisc::testGlobalInstances PASSED
tests/test_configargparse.py::TestMisc::testGlobalInstances_WithName PASSED
It's unclear to me why these exceptions are shown so specifically. I find it confusing, and can't seem to find a way to suppress these warnings.
ResourceWarning: unclosed file
When running python setup.py test, mock generate a ResourceWarning: unclosed file for three tests. The test itself succeeds.
testBasicCase2 (tests.test_configargparse.TestBasicUseCases) ...
/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/traceback.py:220: ResourceWarning: unclosed file <_io.TextIOWrapper name='/var/folders/zx/y85xrx5x0lz6bgqd45lhr9kc0000gp/T/tmpqe8s6ado' mode='r' encoding='UTF-8'>
tb.tb_frame.clear()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
testBasicCase2_WithGroups (tests.test_configargparse.TestBasicUseCases) ...
/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/traceback.py:220: ResourceWarning: unclosed file <_io.TextIOWrapper name='/var/folders/zx/y85xrx5x0lz6bgqd45lhr9kc0000gp/T/tmpwocvu164' mode='r' encoding='UTF-8'>
tb.tb_frame.clear()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
testKwrgsArePassedToArgParse (tests.test_configargparse.TestMisc) ...
/Users/freek/Library/Python/3.8/lib/python/site-packages/mock/mock.py:2072: ResourceWarning: unclosed file <_io.FileIO name=3 mode='rb+' closefd=True>
setattr(_type, entry, MagicProxy(entry, self))
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/Users/freek/Library/Python/3.8/lib/python/site-packages/mock/mock.py:2072: ResourceWarning: unclosed file <_io.FileIO name=4 mode='rb+' closefd=True>
setattr(_type, entry, MagicProxy(entry, self))
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/Users/freek/Library/Python/3.8/lib/python/site-packages/mock/mock.py:2072: ResourceWarning: unclosed file <_io.FileIO name=5 mode='rb+' closefd=True>
setattr(_type, entry, MagicProxy(entry, self))
ResourceWarning: Enable tracemalloc to get the object allocation traceback
The failure of testConfigOrEnvValueErrors is because args is not set when calling TestCase.assertParseArgsRaises. This causes args to be set to sys.args. So calling this test with pytest -k testConfigOrEnvValueErrors -v would give a parser error "unrecognized arguments: -k testConfigOrEnvValueErrors". That clearly is not intended: these tests should not depend on how the test suite was called. I'll make a PR.
Still looking into the other issues, since I still get some unexpected result.
I finally understand what happened. The argparse module contains a function called test_main() which runs all tests. pytest thinks this is a test function itself and calls it as such, which runs all tests again, and also produced the weird stdout of testGlobalInstances that I've seen.
Thanks to the way configargparse is importing tests_argparse, it is possible to rename both TestHelpFormattingMetaclass and test_main() to prevent this behaviour.
The ResourceWarning: unclosed file for testBasicCase2 and testBasicCase2_WithGroups are caused by a minor issue in ArgumentParser._open_config_files().
ArgumentParser._open_config_files() opens multiple configuration files, and returns a list of the file streams. If one of the file open() operations fails, an exception is raised. E.g. file not found.
However, this never returns nor closes the previously opened files, leaving dangling file pointers. This commit fixes the issue by closing previously opened config files when such an exception occurs.
And the ResourceWarning: unclosed file in testKwrgsArePassedToArgParse and testGlobalInstances_WithName did not originate there, but in the test(s) that were run just before that: testConstructor_WriteOutConfigFileArgs and friends.
In testConstructor_WriteOutConfigFileArgs, testConstructor_WriteOutConfigFileArgs2 and testConstructor_WriteOutConfigFileArgsLong, a temporary file cfg_f is created that was never closed.
And with that, I think I caught all errors and warnings in the test suite, so I can make a PR and start adding my own tests (knowing if any issues occur, it is because of my code and not existing code). I'll try to do that later today.
All above issues are addressed in PR #214.
Since #214 is merged, this can be closed.