Giters

python-poetry / poetry

Python packaging and dependency management made easy

Home Page:https://python-poetry.org

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

JSONDecodeError on Python 3.10

robd003 opened this issue · comments

commented

Issue

Poetry crashes whenever it evaluates a package for installation. Running poetry install will return JSONDecodeError 100% of the time.

$ poetry install
Installing dependencies from lock file

Package operations: 16 installs, 0 updates, 0 removals

  • Installing sanic-routing (0.6.2): Failed

  JSONDecodeError

  Expecting value: line 1 column 1 (char 0)

  at /usr/local/Cellar/python@3.10/3.10.0b3/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/decoder.py:355 in raw_decode
      351│         """
      352│         try:
      353│             obj, end = self.scan_once(s, idx)
      354│         except StopIteration as err:
    → 355│             raise JSONDecodeError("Expecting value", s, err.value) from None
      356│         return obj, end
      357│
commented

I've tried removing the entire pypoetry cache directory and still having the same issue

commented

Just tried the 1.2.0a1 release and it's working fine.

commented

The 1.1.7 update pushed out today still has the same problem on Python 3.10

commented

Reproducible with 3.10 on Linux & Windows as well.

commented

BTW disabling poetry's experimental new installer may be a workaround for now:

$ poetry config experimental.new-installer false
commented

I can confirm that the issue is resolved by @hoefling's suggestion.

BTW disabling poetry's experimental new installer may be a workaround for now:

$ poetry config experimental.new-installer false
commented

I can confirm I can reproduce with version 1.1.8 here. Poke @python-poetry/triage

commented

And this is fixed in 1.2.0a2. I guess we can close this ?

commented

I am glad I found this issue. I wouldn't have known how to even debug this, since Poetry eats the whole traceback I only shows this JSONDecodeError.

Is there a way to print the traceback?

commented

Anyone got this on 3.9.7? I get

 • Installing pycparser (2.20): Failed

  JSONDecodeError

  Expecting value: line 1 column 1 (char 0)

  at /usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py:355 in raw_decode
commented

Same here, with a project with both 3.9 and 3.10 env. Removing the 3.10 env solve the problem. However, 3.10 will be released in 2 days, so I don't think we should be surprise there are still things that don't work yet.

commented

Reproduced with Poetry 1.1.10 installed on Python 3.9.5 (Ubuntu 21.04 system Python) using pipx.

poetry install started failing immediately when I switched project from Python 3.9.7 to 3.10.0rc2, which is due for final release tomorrow.

Works:

pyenv local 3.9.7
poetry env use $(pyenv which python)
poetry install

Fails:

pyenv local 3.10.0rc2
poetry env use $(pyenv which python)
poetry install

Poetry eats most of the stack trace. I see the following repeated once for each dependency:

  • Installing urllib3 (1.26.7): Failed

  JSONDecodeError

  Expecting value: line 1 column 1 (char 0)

  at /usr/lib/python3.9/json/decoder.py:355 in raw_decode
      351│         """
      352│         try:
      353│             obj, end = self.scan_once(s, idx)
      354│         except StopIteration as err:
    → 355│             raise JSONDecodeError("Expecting value", s, err.value) from None
      356│         return obj, end
      357│

Removing the Poetry cache with rm -r ~/.cache/pypoetry/cache/ or m -r ~/.cache/pypoetry/ does not improve the situation.

Disabling the experimental installer with poetry config experimental.new-installer false does resolve the issue, so that at least serves as an indication of what part of Poetry is failing.

commented

same issue with python 3.10-rc2 while 3.10 is supposed to be released today. python 3.9 worked well.

with 3.10-rc2, poetry config experimental.new-installer false can make this error disappear.

commented

Hi all,

Wanted to update you that given error should gone with latest poetry==1.1.11 release.

With given release you don’t need to disable experimental new installer to install your dependencies with poetry at 3.10.0rc2 or 3.10.0.

commented

Confirmed for me, fresh stable 3.10 works with latest poetry.

Be careful when upgrading, there are packages that don't have a wheel yet, and you'll need to compile them while in 3.9, you didn't need to.

For me, you can close the ticket.

commented

This issue also occurs on Python3.6 with:

  • Poetry version: 1.1.11
  • Poetry-core version: 1.0.7
  • OS version and name: Fedora 34

The two workarounds mentioned in this ticket work:

  1. Clearing the poetry cache
  2. Disabling the experimental installer using poetry config experimental.new-installer false
commented

Came here to say it started happening out of nowhere on poetry 1.1.11 with python 3.8.12 on github actions ubuntu 20.04 when I added beautifulsoup4 as a dev dependency. No idea what makes that package special, I cleared the cache in the meantime.

commented

@IceTDrinker same here for me. so confused how this could have just started happening.

commented

Can someone provide steps to reproduce this, trying to addd bs4 as dev dependency in a python:3.8.12 container with poetry@1.1.11 did not reproduce the issue. Ideally somehting I can reproduce inside a python container.

commented

Can someone provide steps to reproduce this, trying to addd bs4 as dev dependency in a python:3.8.12 container with poetry@1.1.11 did not reproduce the issue. Ideally somehting I can reproduce inside a python container.

I'll try to see if I can give you a minimal repro, but for me it started happening after the bs4 package was added and the cache from a previous github actions run was used.

Edit : it is run in a container based on ubuntu 20.04 where python is installed thanks to the setup python action. Poetry is installed with pip

commented

I've started getting this issue intermittently, when running poetry update, add or install. The commands fail on some of the dependencies, but not always the same; re-running the command usually results in one or two failures (with different packages each time), until it works usually on the third attempt or so.

commented 
poetry cache clear --all pypi
poetry cache clear --all private

did the trick. For me which python version it run on seemed irrelevant.

commented

@berislavlopac thank you! I re-ran a plugin add a few times and finally it worked, have the same JSON decode error usually when running:

poetry plugin add "keyrings.google-artifactregistry-auth"

commented

Can reproduce on Windows.
Windows 10
Python 3.7.9
Poetry 1.1.11
pip 21.3.1

Previous workaround by reassigning the poetry cache $env:POETRY_CACHE_DIR='.\poetrycache' did not work.
Workaround clearing the cache by doing poetry cache clear --all pypi did seem to work.

commented

Try installing CacheControl==0.12.6 or wait for psf/cachecontrol#264

What happens:

  1. After running poetry install at the first time every pypi.org response is cached. Besides, pypi.org sets cache control, its says max age for cached response is 900 secs
  2. Wait some time (>900 secs)
  3. Run poetry update. At this point cachecontrol trying to use the data from the cache, but its stale. So it asks pypi.org for new response, dumps it and returns to the requests library, pretending that its genuine. But when it dumps, it consumes buffer and dont rewind. So, when we do response.json(), requests lib is reading the buffer, but sinces its already consumed, what it gets is ''. Thats leading to JsonDecodeError.

What happens when you disable experimentail installer - the same thing as running poetry update second time. Cache was updated at step 3, so, installation process continues to the next stale item.

commented

The error is resolved with experimental.new-installer false every time.
With experimental.new-installer true, sometimes the error JSONDecodeError is raised and it works in next attempt without changing experimental.new-installer value. I guess it matches with ddc67cd's comment.

I tested with:

$ ./working_venv/bin/poetry config --list | grep new-installer
experimental.new-installer = true

#! /bin/sh

VENV_DIR="venv"
PYTHON='python3.9'

rm -rf "${VENV_DIR}"
${PYTHON} -m venv "${VENV_DIR}"

"./${VENV_DIR}/bin/pip" install --upgrade pip setuptools poetry

"./${VENV_DIR}/bin/poetry" i

pyproject.toml file:

[tool.poetry]
name = "demo"
version = "0.1.0"
description = "demo"
authors = ["<demo@domain.tld>"]

[tool.poetry.dependencies]
Markdown = "^3.*"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

It can fail with other dependencies than Markdown.

There was a poetry.toml file during the tests but I don't think it's relevant:

[virtualenvs]
create = false
commented

So, the issue is resolved with the new version of cachecontrol==0.12.9 (it will be installed automatically).

Besides, when trying to reproduce this bug in a fresh container I see "SolverProblemError", thus, after this update some of the issues with that error could be resolved as well.

How to reproduce:

docker run --rm -ti python:3.9 bash
pip install poetry CacheControl==0.12.8
poetry init -n
poetry config experimental.new-installer false
poetry add asyncpg


wait 900 secs


poetry update  <- fails
poetry update  <- succees
commented

I run into this issue on Python 3.8.8 as well.

commented

Here's how to fix it:

$ pip install -U cachecontrol
commented

I use rm -rf ~/.cache/pypoetry/cache/.

commented

Hello everyone,

as others already mention this issue is solved by a new version of cachecontrol.

If you install poetry by the install-poetry.py scrip, the new version is installed automatically.

If you use the deprecated get-poetry.py script, it is not possible to update the cachecontrol version. get-poetry.py vendores the dependencies based on poetry's lock file. get-poetry.py installs poetry 1.1 which still supports python2. The current version of cachecontrol doesn't support python2 anymore, thus we cannot update the locked version in the 1.1 branch.

TL;DR;: Install poetry with the install-poetry.py to resolve this issue.

fin swimmer

commented

fresh install with install-poetry.py, deleting poetry.lock, and latest version of cachecontrol did not fix the issue for me, FYI

commented

The problem for me is that something was spewing "Cannot find the file specified" when launching python. Poetry initializes by running python and printing "sys.path" and then attempts to decode this using the json parser.

The JSON parser fails (loads) because the json string starts with "Cannot find the file specified", and of course that's not valid json.

Still a total mystery what is spewing that string though

commented

Installed poetry using pip on Windows on 3.9.6, cannot get poetry install to work :( Tried all suggesitons in this post, with no luck

commented

Installed poetry using pip on Windows on 3.9.6, cannot get poetry install to work :( Tried all suggesitons in this post, with no luck

This bug is closed. If you have a new bug, please create a new bug report, with all the details requested (inputs, outputs, errors, exactly what commands you ran).

commented

is this bug fixed ? im running a fresh conda install on Windows WSL Ubuntu with Conda and Python 3.10 just now. I had the exact same issue. Only thing that worked was poetry config experimental.new-installer false

commented

If anyone else gets here: my issue was not solved by changing config, nor by a complete reinstall of poetry.

Solution: Simply delete and recreate my currently activated virtual environment.

Explanation: as @jcowles suggests above, my currently activated virtual environment python was semi-borked. It was, unbeknownst to me, dishing out a nice error message (but still actually functioning fine somehow):

% python
Error processing line 1 of /Users/james/miniconda/envs/mamod/lib/python3.7/site-packages/distutils-precedence.pth:

  Traceback (most recent call last):
    File "/Users/james/miniconda/envs/mamod/lib/python3.7/site.py", line 168, in addpackage
      exec(line)
    File "<string>", line 1, in <module>
  ModuleNotFoundError: No module named '_distutils_hack'

Remainder of file ignored
Python 3.7.13 (default, Mar 28 2022, 07:24:34)
[Clang 12.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

This looks to be related to pypa/setuptools#2957. Presumably I had broken my venv via various installations.

So my advice if you get a JSONDecodeError with poetry is to simply execute python, and see if you get a weird error and, if so, just trash and recreate your venv.

@clintonroy - we could raise a feature request for poetry to handle the JSON parsing more gracefully? I believe this is done within this file: https://github.com/python-poetry/poetry/blob/ae03bdcdecbb00f2691dbd0f78c08fc010e87779/src/poetry/utils/env.py

commented

I just had the same issue. I tried first deleting the virtual environment (as per @jamesowers-roo), but that didn't fix it.

What worked was the solution from @hoefling:

BTW disabling poetry's experimental new installer may be a workaround for now:

$ poetry config experimental.new-installer false
commented

The JSONDecodeError discussed here is very specifically related to cache-control. Other JSONDecodeErrors are likely caused by broken systems/virtual environments and are more general (not Windows) versions of #6664.

commented

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.