Unable to import local module for custom handlers

[Will Fleming]

Thumbor request URL

N/A

Expected behaviour

Should be able to import a local module for custom handlers, as documented at https://thumbor.readthedocs.io/en/stable/custom_handler_lists.html.

Actual behaviour

ModuleNotFoundError: No module named 'foo'

> thumbor --conf=./thumbor.conf
Traceback (most recent call last):
  File "/usr/local/bin/thumbor", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/thumbor/server.py", line 158, in main
    importer = get_importer(config)
               ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/thumbor/server.py", line 63, in get_importer
    importer.import_modules()
  File "/usr/local/lib/python3.11/site-packages/thumbor/importer.py", line 90, in import_modules
    self.import_item("HANDLER_LISTS", is_multiple=True)
  File "/usr/local/lib/python3.11/site-packages/thumbor/importer.py", line 135, in import_item
    self.load_multiple_item(
  File "/usr/local/lib/python3.11/site-packages/thumbor/importer.py", line 172, in load_multiple_item
    module = self.import_class(
             ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/thumbor/importer.py", line 56, in import_class
    kls = import_class(name, get_module)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/thumbor/importer.py", line 20, in import_class
    module = import_module(module_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1142, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'foo'

Operating system

Debian 11 (python:3-slim docker image)

Your thumbor.conf

from thumbor.handler_lists import BUILTIN_HANDLERS
HANDLER_LISTS = BUILTIN_HANDLERS + ['foo']

(Not my real full .conf, of course, but reproduces the issue.)

foo.py in the current working directory (same directory as thumbor.conf)

from typing import Any, cast
from thumbor.handler_lists import HandlerList
from thumbor.handlers import BaseHandler

class IndexHandler(BaseHandler):
    async def get(self):
        self.set_status(302)
        self.set_header("Location", "https://www.example.com")

def get_handlers(_context: Any) -> HandlerList:
    return [ ("/", IndexHandler) ]

Additional details

Thumbor 7.5.0

python -c 'import foo' or import foo in a repl both work correctly, and I can call foo.get_handlers(None) in a repl and get the expected list.

I added some logging and pdb breakpoints locally, and inspecting the state at https://github.com/thumbor/thumbor/blob/master/thumbor/importer.py#L20 just before the failing import, I can see that os.getcwd() is still the directory I expect, where foo.py & thumbor.conf are. Oddly, the <class '_frozen_importlib_external.PathFinder'> entry in sys.meta_path at this point fails to find the module (sys.meta_path[i].find_module('foo') returns None, while in a python repl it successfully finds the module).

I presume that's the cause here, but I'm unclear on what about the execution environment changes between a plain python repl & the thumbor binary that is changing the import behavior. I am not setting PYTHONPATH or anything explicitly, and skimming through code I can't find anything in thumbor that seems to intentionally be changing the import paths. Is having a custom handler list in a local file like this intended to work, and is there some config I'm missing to make this work? Thanks for your help.

[Will Fleming]

Circling back here: I found that setting PYTHONPATH to the directory of my thumbor config/module source fixed the issue. This surprised me a bit since docs didn't mention this as necessary and no env change was necessary to load the module from a repl. Not sure if this is expected behavior docs fail to mention or not. I'll leave this open since it seems like maybe there is at minimum a docs bug here, but if anyone else with this issue stumbles across this at least setting PYTHONPATH is a workaround.

[Pablo Aguiar]

Hi, this is a known Python behavior and is documented here: https://docs.python.org/3/reference/import.html#path-entry-finders