Tutor claims to support Python 3.12, but apparently doesn't (if plugins are involved)

[Florian Haas]

Bug description

In the setup.py classifiers, Tutor claims to support Python 3.12. But that doesn't appear to be the case, at least not when one uses plugins, apparently due to the continued use of the pkg_resources module in the plugin loading framework.

How to reproduce

• Create a virtualenv using python3.12 as its interpreter.
• Run pip install tutor==17.0.0
• Install a plugin that supports Tutor 17.
• Run tutor plugins enable <plugin>

Then:

  py312: commands[0]> tutor plugins enable s3
  Traceback (most recent call last):
    File "/home/runner/work/tutor-contrib-s3/tutor-contrib-s3/.tox/py312/bin/tutor", line 5, in <module>
      from tutor.commands.cli import main
    File "/home/runner/work/tutor-contrib-s3/tutor-contrib-s3/.tox/py312/lib/python3.12/site-packages/tutor/commands/cli.py", line 11, in <module>
      from tutor.commands.config import config_command
    File "/home/runner/work/tutor-contrib-s3/tutor-contrib-s3/.tox/py312/lib/python3.12/site-packages/tutor/commands/config.py", line 9, in <module>
      from tutor import config as tutor_config
    File "/home/runner/work/tutor-contrib-s3/tutor-contrib-s3/.tox/py312/lib/python3.12/site-packages/tutor/config.py", line 7, in <module>
      from tutor import env, exceptions, fmt, hooks, plugins, serialize, utils
    File "/home/runner/work/tutor-contrib-s3/tutor-contrib-s3/.tox/py312/lib/python3.12/site-packages/tutor/env.py", line 10, in <module>
      import pkg_resources
  ModuleNotFoundError: No module named 'pkg_resources'
  py312: exit 1 (0.17 seconds) /home/runner/work/tutor-contrib-s3/tutor-contrib-s3> tutor plugins enable s3 pid=2021

This was taken from this build log:
https://github.com/hastexo/tutor-contrib-s3/actions/runs/7273329871/job/19817088029?pr=68#step:5:46

The offending line is here:
https://github.com/overhangio/tutor/blob/v17.0.0/tutor/env.py#L10

Additional context

As far as I understand, this is due to python/cpython#95299.
I believe the way to make pkg_resources available to Tutor again would be to add setuptools to the install_requires list in setup.py.

This issue is being masked in production environments by the fact that a command to install setuptools is included in the Tutor openedx Dockerfile. So even if that Dockerfile switched to installing Python 3.12 (rather than 3.8, which it does now), most Tutor users would be unaffected. But plugin authors, who would presumably want to run their integration tests without the full Dockerfile, would very much benefit from the install_requires list being corrected directly in setup.py.

[Abdul-Muqadim-Arbisoft]

Proposed Solution:
Replace pkg_resources with importlib.resources and importlib.metadata:

importlib.resources: This module, included in Python's standard library, provides support for accessing files and resources within packages. It is a suitable replacement for similar functionalities previously achieved through pkg_resources.
importlib.metadata: This module allows access to metadata of Python packages and is a modern alternative to pkg_resources for metadata handling.
Update All Plugin Code:

Many plugins in Tutor might be dependent on pkg_resources. These plugins will require refactoring to adapt to the new importlib modules. This update is crucial for ensuring the seamless functioning of Tutor and its plugins under Python 12.
Temporary Addition of setuptools to install_requires:Since in future pkg resources would also be removed from the setup tools as well

As an interim solution, it's proposed to add setuptools to the install_requires section of Tutor. This step will ensure the availability of pkg_resources until the full transition is completed. It's a temporary measure to maintain stability and compatibility during the transition phase.

[Régis Behmo]

In tutor core, pkg_resources is used for two things:

for entrypoint in pkg_resources.iter_entry_points(...)

and

TEMPLATES_ROOT = pkg_resources.resource_filename(...)

How should these pieces of code be updated if we were to switch to importlib?

[Abdul-Muqadim-Arbisoft]

Iterating Over Entry Points:
The pkg_resources.iter_entry_points(...) is often used to iterate over entry points specified in the package's setup.py. With importlib, we would use importlib.metadata.entry_points().

from importlib.metadata import entry_points
for entrypoint in entry_points().select(group='group'):

Getting Resource Filenames:
The pkg_resources.resource_filename(...) function is used to get the path to a resource within a package. In importlib, we would use importlib.resources.path.

from importlib import resources
with resources.path('package', 'resource') as resource_path:
TEMPLATES_ROOT = str(resource_path)

[Régis Behmo]

Looking good. And are these importlib functions compatible with Python 3.8?

[Abdul-Muqadim-Arbisoft]

importlib.metadata (entry points):
(Available in Python 3.8 and later.)

importlib.resources (resource filenames):
(Available in Python 3.7 and later.)

[Régis Behmo]

Awesome! Then please go ahead and get rid of pkg_resources :)

[Florian Haas]

Just an explanatory note here, for posterity.

In Python 3.8 one could, in principle, use importlib.resources from the standard library, to which it was added in Python 3.7.

However, before Python 3.8, importlib.resources in the stdlib does not have a files attribute (it was added in 3.9).

Thus, as long as Tutor wants to support Python 3.8, importlib_resources must be installed as an external dependency. Once Tutor ceases to support Python 3.8, the external dependency on importlib_resources can be dropped, and importlib.resources from the standard library used instead.