Installing praw from source or sdist fails due to importing runtime deps at build time

Question

Installing praw from source or sdist fails due to importing runtime deps at build time

CAM-Gerlach opened this issue 5 months ago · comments

Describe the Bug

Same issue as praw-dev/prawcore#164 and @LilSpazJoekp requested followup from praw-dev/prawcore#165

Due to the use of Flit's fallback automatic metadata version extraction needing to dynamically import the package __init__.py instead of reading it statically (see pypa/flit#386 ) , since the actual __version__ is imported from const.py, this requires the runtime dependencies be installed in the build environment when building or installing from an sdist (or the source tree).

This happens merely by chance to currently be a transitive backend dependency of flit_core and thus building in isolated mode works. However, this shouldn't be relied upon, as if either flit_core or praw's dependencies were to ever change, this would break isolated builds too. And requests isn't an exposed non-backend dependency of flit-core, so it doesn't actually get installed otherwise.

This means Requests must be manually installed (not just the explicitly pyproject,toml-specified build dependencies) if building/installing praw in any other context than pip's isolated mode, e.g. without the --no-build-isolation flag, via the legacy non-PEP 517 builder, or via other tools or most downstream ecosystems that use other build isolation mechanisms. In particular, this was an issue on conda-forge/prawcore-feedstock#14 where I was packaging the new prawcore 2.4.0 version for Conda-Forge—you can see the full Azure build log.

This can be worked around for now by manually installing requests into the build environment, but that's certainly not an ideal solution as it adds an extra build dependency (and its dependencies in turn), requires extra work by everyone installing from source (without build isolation), makes builds take longer, and is fragile and not easily maintainable/scalable long term for other runtime dependencies.

Desired Result

Praw is able to be built and installed without installing requests, or relying on it happening to be a transitive build backend dependency of flit_core and present "by accident".

While there are other ways to achieve this, as described in praw-dev/prawcore#164 , @LilSpazJoekp indicated he preferred moving __version__ to be in __init__.py directly, so Flit can find it by static inspection without triggering an import, and naturally asked for the same approach here as applied in praw-dev/prawcore#165 .

This works, with one additional complication: the USER_AGENT_FORMAT module-level constant in const.py relies in turn on __version__, which means that praw.const needs to import it from praw (__init__), and because praw.reddit.Reddit is imported in __init__ for convenience (which in turn imports praw.const), this creates an import cycle (as well as making praw expensive to import, on the order of multiple seconds on an older machine).

This cannot be trivially solved by a scope-local import, because USER_AGENT_FORMAT is a module-level constant. The simplest and likely least disruptive solution, since USER_AGENT_FORMAT is only used on place in PRAW (praw.reddit) is to just inject __version__ at the one point of use along with the user-configured user agent rather than statically, eliminating an extra layer of format replacement nesting in the process.

The one non-trivial impact from this is requiring a change to user code directly using USER_AGENT_FORMAT themselves to insert praw.__version__ along with their own user agent. After various grep.app queries, which scans the entirety of public code on GitHub, I did find a single instance of this that would be affected, in the (closed-development) nnreddit project providing a Reddit backend for the Gnus reader. However, the fix is straightforward (an extra arg to one format call) and backward-compatible with older PRAW, and the primary change here, removing, __version__ from praw.const is also equally technically backward-incompatible and was accepted in prawcore, so not sure how much of an issue that is for you.

Using a named placeholder for the PRAW version, e.g."{} PRAW/{praw_version}", would give a clearer error message as to what is needed to add to the format call, while modifying existing code (e.g. USER_AGENT_FORMAT.format(user_agent)) to support the new PRAW version (i.e. USER_AGENT_FORMAT.format(user_agent, praw_version=praw.__version__) would be fully backward compatible with USER_AGENT_FORMAT in older PRAW, as format will simply drop arguments that don't match placeholders in the string.

The other alternative as mentioned in the other issue would be simply setting the version statically in pyproject.toml. This would avoid all these issues and needing to change anything else besides the set_version.py script as well as give you the benefits of static metadata. It would mean the version is defined multiple places which you wanted to avoid, but if you're using that script to set it anyway, it shouldn't actually require any additional maintainer effort since the script could take care of it too. Of course, if that's the case we'd presumably want to go back and make that same change in prawcore too; however, it would simplify the implementation in asyncpraw (as well as asyncprawcore) and avoid this same issue with USER_AGENT_FORMAT, and for which asyncpraw.const.__version__ is used at least once in the wild.

Up to you!

Code to reproduce the bug

In a fresh venv without requests or praw installed, e.g.

python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip wheel

Run:

pip install flit-core  # Or flit
pip install --no-build-isolation .

My code does not include sensitive credentials

Yes, I have removed sensitive credentials from my code.

Relevant Logs

See praw-dev/prawcore#164

This code has previously worked as intended

Yes

Operating System/Environment

Any

Python Version

Any, tested 3.9-3.12

PRAW Version

7.7.2.dev0

Links, references, and/or additional comments?

No response

Joel Payne · Answer 1 · Mon Dec 18 2023 07:40:44 GMT+0800 (China Standard Time)

When I run your reproductions steps I'm not encountering this issue with requests being required for install.

Here's what I'm doing:

Setup up the Python virtual environment.

python3.12 -m venv .venv
source .venv/bin/activate
pip install -U pip

Run code to reproduce error using local praw repo.
Command:

pip install flit-core
pip install --no-build-isolation . --no-binary .

Output:

Collecting flit-core
  Using cached flit_core-3.9.0-py3-none-any.whl.metadata (822 bytes)
Using cached flit_core-3.9.0-py3-none-any.whl (63 kB)
Installing collected packages: flit-core
Successfully installed flit-core-3.9.0
Processing /Users/jkpayne/PythonProjects/praw
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [27 lines of output]
      Traceback (most recent call last):
        File "/Users/jkpayne/PythonProjects/praw/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/Users/jkpayne/PythonProjects/praw/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/jkpayne/PythonProjects/praw/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 149, in prepare_metadata_for_build_wheel
          return hook(metadata_directory, config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/jkpayne/PythonProjects/praw/.venv/lib/python3.12/site-packages/flit_core/buildapi.py", line 49, in prepare_metadata_for_build_wheel
          metadata = make_metadata(module, ini_info)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/jkpayne/PythonProjects/praw/.venv/lib/python3.12/site-packages/flit_core/common.py", line 425, in make_metadata
          md_dict.update(get_info_from_module(module, ini_info.dynamic_metadata))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/jkpayne/PythonProjects/praw/.venv/lib/python3.12/site-packages/flit_core/common.py", line 222, in get_info_from_module
          docstring, version = get_docstring_and_version_via_import(target)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/jkpayne/PythonProjects/praw/.venv/lib/python3.12/site-packages/flit_core/common.py", line 195, in get_docstring_and_version_via_import
          spec.loader.exec_module(m)
        File "<frozen importlib._bootstrap_external>", line 994, in exec_module
        File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
        File "/Users/jkpayne/PythonProjects/praw/praw/__init__.py", line 12, in <module>
          from .reddit import Reddit
        File "/Users/jkpayne/PythonProjects/praw/praw/reddit.py", line 15, in <module>
          from prawcore import (
      ModuleNotFoundError: No module named 'prawcore'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Install prawcore from local prawcore repo
Command:

pip install --no-build-isolation ../prawcore --no-binary ../prawcore

Output:

Processing /Users/jkpayne/PythonProjects/prawcore
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting requests<3.0,>=2.6.0 (from prawcore==2.4.1.dev0)
  Using cached requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
Collecting charset-normalizer<4,>=2 (from requests<3.0,>=2.6.0->prawcore==2.4.1.dev0)
  Using cached charset_normalizer-3.3.2-cp312-cp312-macosx_10_9_x86_64.whl.metadata (33 kB)
Collecting idna<4,>=2.5 (from requests<3.0,>=2.6.0->prawcore==2.4.1.dev0)
  Using cached idna-3.6-py3-none-any.whl.metadata (9.9 kB)
Collecting urllib3<3,>=1.21.1 (from requests<3.0,>=2.6.0->prawcore==2.4.1.dev0)
  Using cached urllib3-2.1.0-py3-none-any.whl.metadata (6.4 kB)
Collecting certifi>=2017.4.17 (from requests<3.0,>=2.6.0->prawcore==2.4.1.dev0)
  Using cached certifi-2023.11.17-py3-none-any.whl.metadata (2.2 kB)
Using cached requests-2.31.0-py3-none-any.whl (62 kB)
Using cached certifi-2023.11.17-py3-none-any.whl (162 kB)
Using cached charset_normalizer-3.3.2-cp312-cp312-macosx_10_9_x86_64.whl (122 kB)
Using cached idna-3.6-py3-none-any.whl (61 kB)
Using cached urllib3-2.1.0-py3-none-any.whl (104 kB)
Building wheels for collected packages: prawcore
  Building wheel for prawcore (pyproject.toml): started
  Building wheel for prawcore (pyproject.toml): finished with status 'done'
  Created wheel for prawcore: filename=prawcore-2.4.1.dev0-py3-none-any.whl size=17264 sha256=6ab3e4d2378eb16941e6d8acc6789916ca4f52a1f63f60aa89148b12b2211e8a
  Stored in directory: /private/var/folders/xf/ldd27nrn7_386m2frx69301c0000gp/T/pip-ephem-wheel-cache-5qgatd7r/wheels/aa/6d/68/a5119784a421be35656aba4aa4b2eba109e01530cb65bc4414
Successfully built prawcore
Installing collected packages: urllib3, idna, charset-normalizer, certifi, requests, prawcore
Successfully installed certifi-2023.11.17 charset-normalizer-3.3.2 idna-3.6 prawcore-2.4.1.dev0 requests-2.31.0 urllib3-2.1.0

Run the code to reproduce error again
Command:

pip install --no-build-isolation . --no-binary .

Output:

Processing /Users/jkpayne/PythonProjects/praw
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [35 lines of output]
      Traceback (most recent call last):
        File "/Users/jkpayne/PythonProjects/praw/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/Users/jkpayne/PythonProjects/praw/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/jkpayne/PythonProjects/praw/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 149, in prepare_metadata_for_build_wheel
          return hook(metadata_directory, config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/jkpayne/PythonProjects/praw/.venv/lib/python3.12/site-packages/flit_core/buildapi.py", line 49, in prepare_metadata_for_build_wheel
          metadata = make_metadata(module, ini_info)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/jkpayne/PythonProjects/praw/.venv/lib/python3.12/site-packages/flit_core/common.py", line 425, in make_metadata
          md_dict.update(get_info_from_module(module, ini_info.dynamic_metadata))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/jkpayne/PythonProjects/praw/.venv/lib/python3.12/site-packages/flit_core/common.py", line 222, in get_info_from_module
          docstring, version = get_docstring_and_version_via_import(target)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/jkpayne/PythonProjects/praw/.venv/lib/python3.12/site-packages/flit_core/common.py", line 195, in get_docstring_and_version_via_import
          spec.loader.exec_module(m)
        File "<frozen importlib._bootstrap_external>", line 994, in exec_module
        File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
        File "/Users/jkpayne/PythonProjects/praw/praw/__init__.py", line 12, in <module>
          from .reddit import Reddit
        File "/Users/jkpayne/PythonProjects/praw/praw/reddit.py", line 28, in <module>
          from . import models
        File "/Users/jkpayne/PythonProjects/praw/praw/models/__init__.py", line 4, in <module>
          from .helpers import DraftHelper, LiveHelper, MultiredditHelper, SubredditHelper
        File "/Users/jkpayne/PythonProjects/praw/praw/models/helpers.py", line 10, in <module>
          from .reddit.draft import Draft
        File "/Users/jkpayne/PythonProjects/praw/praw/models/reddit/draft.py", line 9, in <module>
          from .subreddit import Subreddit
        File "/Users/jkpayne/PythonProjects/praw/praw/models/reddit/subreddit.py", line 15, in <module>
          import websocket
      ModuleNotFoundError: No module named 'websocket'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Change websockets import to be a local import.

Run the code to reproduce error again. No error occurs.
Command:

pip install --no-build-isolation . --no-binary .

Output:

Processing /Users/jkpayne/PythonProjects/praw
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: prawcore<3,>=2.1 in ./.venv/lib/python3.12/site-packages (from praw==7.7.2.dev0) (2.4.1.dev0)
Collecting update_checker>=0.18 (from praw==7.7.2.dev0)
  Using cached update_checker-0.18.0-py3-none-any.whl (7.0 kB)
Collecting websocket-client>=0.54.0 (from praw==7.7.2.dev0)
  Using cached websocket_client-1.7.0-py3-none-any.whl.metadata (7.9 kB)
Requirement already satisfied: requests<3.0,>=2.6.0 in ./.venv/lib/python3.12/site-packages (from prawcore<3,>=2.1->praw==7.7.2.dev0) (2.31.0)
Requirement already satisfied: charset-normalizer<4,>=2 in ./.venv/lib/python3.12/site-packages (from requests<3.0,>=2.6.0->prawcore<3,>=2.1->praw==7.7.2.dev0) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in ./.venv/lib/python3.12/site-packages (from requests<3.0,>=2.6.0->prawcore<3,>=2.1->praw==7.7.2.dev0) (3.6)
Requirement already satisfied: urllib3<3,>=1.21.1 in ./.venv/lib/python3.12/site-packages (from requests<3.0,>=2.6.0->prawcore<3,>=2.1->praw==7.7.2.dev0) (2.1.0)
Requirement already satisfied: certifi>=2017.4.17 in ./.venv/lib/python3.12/site-packages (from requests<3.0,>=2.6.0->prawcore<3,>=2.1->praw==7.7.2.dev0) (2023.11.17)
Using cached websocket_client-1.7.0-py3-none-any.whl (58 kB)
Building wheels for collected packages: praw
  Building wheel for praw (pyproject.toml): started
  Building wheel for praw (pyproject.toml): finished with status 'done'
  Created wheel for praw: filename=praw-7.7.2.dev0-py3-none-any.whl size=189356 sha256=c0a0cb68afc5accf5a96847c1367aebfbd068dc2e51472fdb5e90950ae43de63
  Stored in directory: /private/var/folders/xf/ldd27nrn7_386m2frx69301c0000gp/T/pip-ephem-wheel-cache-pf5rcd8_/wheels/03/25/df/2a1cafce15246fae04b8bdc466629219cc0de6e6fd95da8b2a
Successfully built praw
Installing collected packages: websocket-client, update_checker, praw
Successfully installed praw-7.7.2.dev0 update_checker-0.18.0 websocket-client-1.7.0

C.A.M. Gerlach · Answer 2 · Tue Dec 19 2023 07:18:31 GMT+0800 (China Standard Time)

Sorry, I copy-pasted a lot of the relevant content from the prawcore issue, and I missed updating some important bits. Namely, the error message will reference whatever dependenc(ies) of the package don't already happen to be installed in the build environment, starting from the first missing import it hits, in this case prawcore. As such, the command (or more simply for this case, pip install --no-build-isolation .) in your Step 2 does repro the error (as you found), just with a different missing dependency (in this case prawcore).

After you manually install prawcore (which works correctly due to the fix we implemented over there, and installs most of the other dependencies including requests), you then see another error due to the missing websockets dependency at build time, required for the same reasons. After patching that out, everything works (evidently because the only other PRAW-unique dependency, update_checker, happens to not be in the import chain triggered by reddit.py in __init__,py currently).

So, TL;DR, the issue is still present as reported, just with a different error message, harder to hack around and more likely to occur (due to the dependencies being much less common to happen to be installed in the build environment due to other build deps or by default). I've updated the title and initial description accordingly, thanks.

github-actions · Answer 3 · Thu Jan 18 2024 09:04:12 GMT+0800 (China Standard Time)

This issue is stale because it has been open for 30 days with no activity.

Remove the Stale label or comment or this will be closed in 30 days.

C.A.M. Gerlach · Answer 4 · Tue Jan 30 2024 15:52:54 GMT+0800 (China Standard Time)

Hey @LilSpazJoekp , have you gotten a chance to give this another look and lmk how you want to move forward here?

Simply using static metadata and having the helper script you already use to update the version other places updated it too is, after a lot of careful thought and testing is the solution I'd recommend, as it is a ≈2 line change that doesn't require touching the code at all and is fully backward-compatible, plus gets you the benefits of standardized static metadata and can be easily applied consistently across all your repos without requiring customization and testing for each.

However, I've also proposed another solution along the lines of what I originally implemented in prawcore, albeit somewhat more complicated and does have a small but non-zero backward compat impact.

Lmk, thanks!

github-actions · Answer 5 · Thu Feb 29 2024 20:24:44 GMT+0800 (China Standard Time)

This issue is stale because it has been open for 30 days with no activity.

Remove the Stale label or comment or this will be closed in 30 days.

github-actions · Answer 6 · Sun Mar 31 2024 02:17:14 GMT+0800 (China Standard Time)

This issue was closed because it has been stale for 30 days with no activity.