`wrap_module` has limitations with relative imports

[Peter Bierma]

Describe the bug

importlib.import_module requires the package argument to be non-null in the case of a relative import (e.g. importlib.import_module("..abc") is not allowed). This causes problems when trying to import through a directory in the wrap_module method. Take the following directory tree as an example:

.
├── module
│   └── conf.py
└── my_config.py

This will cause a TypeError when trying to call wrap_module to import my_config in configzen, because the top level module directory is not a python package, causing the passed value to import_module to be None.

To reproduce

# the error is relative to the filesystem, so heres a quick script that would generate a repro
import os

os.mkdir("./repro")
os.mkdir("./repro/module")

with open("./repro/module/conf.py", "w") as f:
    f.write(
        """from configzen import ConfigModel

class Test(ConfigModel):
    host: str
    port: int


Test.wrap_module(".my_config")"""
    )

with open("./repro/my_config.py", "w") as f:
    f.write(
        """host = '0.0.0.0'
port = 5000"""
    )

print("now try python3 repro/module/conf.py")

Error

Traceback (most recent call last):
  File "/home/zero/Desktop/Projects/Python/repro/module/conf.py", line 8, in <module>
    Test.wrap_module(".my_config")
  File "/home/zero/.local/lib/python3.10/site-packages/configzen/model.py", line 2430, in wrap_module
    importlib.import_module(module_name, package=package)
  File "/usr/lib/python3.10/importlib/__init__.py", line 121, in import_module
    raise TypeError(msg.format(name))
TypeError: the 'package' argument is required to perform a relative import for '.my_config'

Versions

• OS: Arch Linux
• Python: 3.10.10

Additional context

No response

[Bartosz Sławecki]

Thanks for reporting the issue.

tl;dr: You cannot perform relative imports directly from the __main__ module. Relative imports are only relative in a package. This is not a limitation nor a bug from configzen, but a natural and correct behavior of Python.

The provided example would not work even if the package argument was not None, because the conf.py module is not in the same directory as the my_config.py module. You would have to import ..my_config instead of .my_config.
I am guessing you meant Test.wrap_module("..my_config") and the further explanation is based on this claim.

When executing repro/module/conf.py directly, it is impossible to relatively import repro/my_config.py and that's just fine, because there is no parent package known to __main__. You could probably, however, absolutely-import it:
Please cd /home/zero/Desktop/Projects/Python/.

1. Change the 8th line in repro/module/conf.py to Test.wrap_module("repro.my_config").
   Execute PYTHONPATH=. && python3 repro/module/conf.py. There should be no error.
2. Change the 8th line in repro/module/conf.py to Test.wrap_module("my_config").
   Execute PYTHONPATH=repo && python3 repro/module/conf.py. There should be no error, too.

(See PYTHONPATH documentation here. I change its value to influence sys.path, the reference list to resolving any imports in Python; usually people would think it's just the current working directory, but it doesn't have to be, that's why sometimes they might stumble across a seemingly impossible path-related issue which can be fixed the fastest way by adjusting PYTHONPATH.)

If you want to wrap my_config.py using a relative import in the described case, you must apply the rule that relative imports are only relative in a package. Hence, you cannot run repro/module/conf.py directly, but you need to import it instead.

Follow these steps:

1. Change the 8th line in repro/module/conf.py to the desired Test.wrap_module("..my_config").
2. Add main.py (or any other name) module in the parent package (parent-parent to repro/module/conf.py), i.e. repro/. Your tree should look like this now:

   .
   ├── module
   │   └── conf.py
   ├── main.py
   └── my_config.py
   

3. In main.py, place

   from repro.module import conf
   
   from repro import my_config  # Enjoy your wrapped module!

4. cd /home/zero/Desktop/Projects/Python/ && export PYTHONPATH=. && python3 repro/main.py.

Of course, that's not all. This structure could be contained inside thousands of directories, where the __main__ package is on top of it. And it would still work, though more readable with relative imports inside the innermost modules then.

I hope that helps. If you have any questions, let me know.

[Bartosz Sławecki]

I'm closing this issue for now, but we may still chat here.