Test fails when function doesn't exist in currently installed package
xkortex opened this issue · comments
pip 19.1.1 from /home/mike/.virtualenvs/ve/lib/python3.6/site-packages/pip (python 3.6)
xdoctest 0.8.3
I have a file repo/pkg/util/caching.py, and I have already run pip install repo/. In that file, I added this function:
def truthy_string(s: str) -> bool:
"""Coerce a string into a bool very aggressively
Examples:
>>> truthy_string('true')
True
"""
s = s.lower()
if s == 'false':
return False
if s == 'true':
return True
return TrueWhen I run xdoctest ~/kw/proj/repo/pkg/util/caching.py, I get
Traceback (most recent call last):
File "/home/mike/.virtualenvs/ve/lib/python3.6/site-packages/xdoctest/doctest_example.py", line 539, in run
checker.check_exception(exc_got, want, runstate)
File "/home/mike/.virtualenvs/ve/lib/python3.6/site-packages/xdoctest/doctest_example.py", line 523, in run
got_eval = eval(code, test_globals)
File "<doctest:/home/mike/kw/proj/repo/pkg/util/caching.py::truthy_string:0>", line rel: 1, abs: 23, in <module>
>>> truthy_string('true')
NameError: name 'truthy_string' is not defined
It looks like it is trying to evaluate truthy_string. I think it is able to import pkg.util.caching, which gives it a false sense of confidence, so to speak, so it runs with the installed version of pkg. Since I am not installed with pip -e, that version is stale.
Unsurprisingly, if I run pip install repo/ again, it works (as now pkg.util.caching.truthy_string exists in my virtualenv)
It probably ought to be evaluating from the current repo codebase. I realize this is pretty tricky but I at least wanted you to be aware of this case.
Edit:
Furthermore, if I create a new file in that package, repo/pkg/util/parse.py, and try to import from there in my other code, without a pip install, it totally goes boom:
DOCTEST STDOUT/STDERR
Traceback (most recent call last):
File "~/.virtualenvs/repo/lib/python3.6/site-packages/xdoctest/utils/util_import.py", line 154,
in _custom_import_modpath
module = import_module_from_name(modname)
File "~/.virtualenvs/repo/lib/python3.6/site-packages/xdoctest/utils/util_import.py", line 308,
in import_module_from_name
module = importlib.import_module(modname)
File "~/.virtualenvs/repo/lib/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'pkg.util.parse'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "~/.virtualenvs/repo/bin/xdoctest", line 10, in <module>
sys.exit(main())
File "~/.virtualenvs/repo/lib/python3.6/site-packages/xdoctest/__main__.py", line 149, in main
config=config, durations=durations)
File "~/.virtualenvs/repo/lib/python3.6/site-packages/xdoctest/runner.py", line 161, in doctest_
module
run_summary = _run_examples(enabled_examples, verbose, config)
File "~/.virtualenvs/repo/lib/python3.6/site-packages/xdoctest/runner.py", line 322, in _run_exa
mples
summary = example.run(verbose=verbose, on_error=on_error)
File "~/.virtualenvs/repo/lib/python3.6/site-packages/xdoctest/doctest_example.py", line 438, in
run
self._import_module()
File "~/.virtualenvs/repo/lib/python3.6/site-packages/xdoctest/doctest_example.py", line 396, in
_import_module
self.module = utils.import_module_from_path(self.modpath, index=-1)
File "~/.virtualenvs/repo/lib/python3.6/site-packages/xdoctest/utils/util_import.py", line 274,
in import_module_from_path
module = _custom_import_modpath(modpath, index=index)
File "~/.virtualenvs/repo/lib/python3.6/site-packages/xdoctest/utils/util_import.py", line 161,
in _custom_import_modpath
raise RuntimeError('\n'.join(msg_parts))
RuntimeError: ERROR: Failed to import modname=pkg.util.parse with modpath=~/kw/proj/repo/pkg/util/parse.py
Caused by: No module named 'pkg.util.parse'
I have run across this issue before. Its trying to convert the package path to a package name, and as you said, when it attempts to import that name it sees the installed version in your PYTHONPATH before it sees the local directory.
Because you explicitly passed it a path to the module, it might be possible to cause xdoctest to prefer the repo path over the install path. This would likely be done by modifying xdoctest.core:package_calldefs, xdoctest.core:_rectify_to_modpath and xdoctest.static:package_modpaths. These three functions work together to map the input to the actual module paths that will be tested.
^ nope
Actually, now that I'm thinking about it, the above functions may be doing the right thing because they are parsing out the doctest statically from the file you are giving it. The problem is actually more likely in the runner where it actually tries to import the code so it can test it. A fix for this might actually need to live in xdoctest.doctest_example:DoctestExample._import_module. Looking at it this is almost certainly the problem:. The existing code looks like this:
def _import_module(self):
"""
After this point we are in dynamic analysis mode, in most cases
xdoctest should have been in static-analysis-only mode.
"""
if self.module is None:
if not self.modname.startswith('<'):
# self.module = utils.import_module_from_path(self.modpath, index=0)
self.module = utils.import_module_from_path(self.modpath, index=-1)Note that the index=0 code is commented out for index=-1 code. I think if you swapped these, then it might work. However, using index=0 can be dangerous if the imported modules mucks with sys.path, which unfortunately a lot of libraries do, hence why the default is -1.
Maybe its reasonable to offer some way of modifying this setting? But it does add complexity to the user interface for a very niche problem. I wouldn't mind adding a hidden semi-undocumented and unsupported flag to change this behavior. Then again, you could also just install with -e, or reinstall the library and test the installed version.