tomboulier / python-import-hell

Example of Nested Imports Re-Importing

Python Import Hell

This provides a toy example explaining how to import into a pre-existing namespace without re-creating already existing module objects.

The use-case is for auto-generated code which is included as part of a package.

Code and Imports

The package has the following structure:

foo/
    __init__.py
    _generated/
        __init__.py
        google/
            baz.py
            bing.py
            __init__.py

The baz module imports the bing module, but (under the autogenerated assumption) it imports via

from google import bing

rather than the absolute import

from foo._generated.google import bing

What is the Problem?

When using code like this, user code may execute something like

from foo._generated.google import bing
from foo._generated.google import baz

which actually imports bing twice (once directly and once via baz).

Rather than re-using the original (as is done for standard Python imports), a brand new module object is created.

Why is this a problem? In the case that import bing adds something (assumed to be unique) to a registry, the second import breaks the uniqueness assumption.

How do we fix it?

We fix it by requiring that the generated imports (e.g. from google import bing) happen as absolute imports via:

from __future__ import absolute_import

i.e. from the google package rather than from the foo._generated.google sub-package.

In addition, we manually patch the existing google package (module object) and alias the foo._generated.google object to agree with it

from __future__ import absolute_import
import sys
import google
google.__path__.append('path/to/local/google')
alternate_key = __name__ + '._generated.google'
if alternate_key in sys.modules:
    raise ImportError(alternate_key, 'has already been imported')
# Re-use the module
sys.modules[alternate_key] = google