OctoPrint 1.9.x can't start on some systems with pydantic 1.10.13+
foosel opened this issue · comments
Problem
On some OctoPrint instances with pydantic==1.10.13 installed (e.g. pulled in while updating to OctoPrint 1.9.2), the server won't be able to start up afterwards due to an error during settings initialization:
Okt 05 06:02:39 octopi octoprint[933]: 2023-10-05 06:02:39,624 - octoprint.startup - CRITICAL - Could not initialize settings manager: Argument 'bases' has incorrect type (expected list, got tuple)
Okt 05 06:02:39 octopi octoprint[933]: 2023-10-05 06:02:39,624 - octoprint.startup - CRITICAL - There was a fatal error starting up OctoPrint.
Okt 05 06:02:39 octopi octoprint[933]: Could not initialize settings manager: Argument 'bases' has incorrect type (expected list, got tuple)
Okt 05 06:02:39 octopi octoprint[933]: There was a fatal error starting up OctoPrint.
See for example https://community.octoprint.org/t/dosnt-start-after-update-1-9-2/54191 and possibly also https://discord.com/channels/704958479194128507/708230190462926898/1159374186456825878
Solution
Downgrading/pinning pydantic to 1.10.12 resolves the issue for now.
Additional info
The corresponding ticket on pydantic's side seems to be pydantic/pydantic#7689
I'm not able to reproduce this on my local dev environment with either Python 3.11 or 3.12, pulling in pydantic-1.10.13-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl. I'm also not able to reproduce it on the "Test install" GitHub Actions workflow that I've just extended to also fire up the server and check the logs for errors.
I can reproduce it when manually installing pydantic==1.10.13 on the latest OctoPi-UpToDate image, which pulls in pydantic-1.10.13-cp39-cp39-linux_armv7l.whl.
The upstream bug indicates that there's been an issue with the binary builds of pydantic using the wrong cython version, and that's apparently causing these problems. What I don't understand is why this only seems to happen on some platforms (and those also seem to differ between us and the also affected Home Assistant project, who ran into this on i686 and x86_64 builds as far as I understood from the associated tickets), however, given that we won't be able to do anything about that, all that we can do here is pin the dependency to 1.10.12 for now and hope things get solved upstream.
@foosel I ran into this a few weeks ago, with a clean install of 1.9.2 in a venv. Strangely, it was resolved by updating from pydantic 1.10.12 to .13. Yesterday I updated to 1.9.3, which downgraded pydantic to .12 and now I have the same error again.
Unfortunately, downgrading to 1.9.2 and upgrading pydantic to 1.10.13 again doesn't fix it anymore.
Python version 3.11.2
Can you please share the exact error output? And also information about your install, so OS, CPU architecture, python version, etc? This is the first time I have heard of this being an issue with .12 and NOT .13 and that makes this problem even more mysterious.
I ran into this a few weeks ago
Next time you run into an issue that requires manual intervention like this, please open a bug report 😅 It would have helped a TON here, I only learned about this even being a problem late last week. And it would also allowed to debug this further so you wouldn't be running into yet another problem yet again with your setup.
Raspbian Bookworm 12.1 on a Raspberry Pi 3B+
Python 3.11.2 installed from Raspbian repo (has protection for using pip into a distro managed installation, that's why I set up the venv)
Trying to start up Octoprint the way it's being done from systemd:
sudo -u octoprint /var/lib/octoprint/bin/python3 -m octoprint serve --config /var/lib/octoprint/.octoprint/config.yaml --basedir /var/lib/octoprint/.octoprint --host 127.0.0.1 --port 5000
2023-10-11 13:06:41,282 - octoprint.startup - CRITICAL - Could not initialize settings manager: Argument 'bases' has incorrect type (expected list, got tuple)
2023-10-11 13:06:41,283 - octoprint.startup - CRITICAL - There was a fatal error starting up OctoPrint.
Could not initialize settings manager: Argument 'bases' has incorrect type (expected list, got tuple)
There was a fatal error starting up OctoPrint.
Trying to manually import the octoprint.settings module:
>>> from octoprint.settings import settings
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/var/lib/octoprint/lib/python3.11/site-packages/octoprint/settings/__init__.py", line 35, in <module>
from octoprint.schema.config import Config
File "/var/lib/octoprint/lib/python3.11/site-packages/octoprint/schema/config/__init__.py", line 9, in <module>
from .access_control import AccessControlConfig
File "/var/lib/octoprint/lib/python3.11/site-packages/octoprint/schema/config/access_control.py", line 11, in <module>
class AccessControlConfig(BaseModel):
File "pydantic/main.py", line 186, in pydantic.main.ModelMetaclass.__new__
TypeError: Argument 'bases' has incorrect type (expected list, got tuple)
If you need any more testing on an affected system, just tag me so I'll get notified.
@edekeijzer could you provide the output of pip freeze in that environment as well?
I'll try to replicate the situation exactly.
Arc-Welder @ file:///tmp/tmp25e681pl/ArcWelderPlugin-master.zip
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
async-timeout==4.0.3
Auto-bed-leveling-expert @ file:///tmp/tmppo9w08x8/Octoprint_ABL_Expert-0.6.zip
Babel==2.12.1
backoff==1.11.1
Backup-Scheduler @ file:///tmp/tmp1oot0qrt/OctoPrint-BackupScheduler-master.zip
Bed-Leveling @ file:///tmp/tmp88ye228j/OctoPrint-Cr10_leveling-master.zip
Bed-Leveling-Wizard @ file:///tmp/tmpso54yybm/OctoPrint-BedLevelingWizard-master.zip
Bed-Visualizer @ https://github.com/jneilliii/OctoPrint-BedLevelVisualizer/archive/1.1.2rc7.zip
blinker==1.6.2
cachelib==0.10.2
certifi==2023.7.22
cffi==1.15.1
charset-normalizer==3.2.0
class-doc==0.2.6
click==8.1.7
colorlog==6.7.0
colorzero==2.0
decorator==5.1.1
Deprecated==1.2.14
distlib==0.3.7
emoji==2.8.0
feedparser==6.0.10
file-read-backwards==3.0.0
filelock==3.12.4
filetype==1.2.0
Flask==2.2.5
Flask-Assets==2.0
flask-babel==3.1.0
Flask-Limiter==3.5.0
Flask-Login==0.6.2
frozendict==2.3.8
future==0.18.3
gpiozero==1.6.2
idna==3.4
ifaddr==0.2.0
importlib-resources==6.1.0
intelhex==2.3.0
itsdangerous==2.1.2
Jinja2==3.1.2
jsonpath-rw==1.4.0
limits==3.6.0
lxml==4.9.3
Markdown==3.4.4
markdown-it-py==3.0.0
MarkupSafe==2.1.3
Marlin-EEPROM-editor @ file:///tmp/tmp81hcyltq/release.zip
Marlin-GCode-Documentation @ file:///tmp/tmpktkdxyux/MarlinGcodeDocumentation-master.zip
mdurl==0.1.2
mock==5.1.0
more-itertools==10.1.0
multidict==6.0.4
netaddr==0.8.0
netifaces==0.11.0
OctoLight-Home-Assistant @ file:///tmp/tmpdw7t_75b/OctoLight_Home-Assistant-master.zip
OctoPrint==1.9.3
OctoPrint-BLTouch @ file:///tmp/tmpemujm1j1/OctoPrint-BLTouch-master.zip
OctoPrint-CustomControlPlugin @ file:///tmp/tmp6j8nedt2/octoprint-customControl-master.zip
OctoPrint-Dashboard @ file:///tmp/tmptfi9pedi/OctoPrint-Dashboard-master.zip
OctoPrint-DeleteMoveAfterPrint @ file:///tmp/tmp25i3w6lj/master.zip
OctoPrint-DetailedProgress @ file:///tmp/tmp2wasf9mx/OctoPrint-DetailedProgress-master.zip
OctoPrint-DisplayZ @ file:///tmp/tmpbidvvqpt/OctoPrint-DisplayZ-master.zip
OctoPrint-Draggable-files @ file:///tmp/tmp_fa8casy/Octoprint-Draggable-Files-main.zip
OctoPrint-DragonOrder @ file:///tmp/tmp17yjomwg/OctoPrint-DragonOrder-master.zip
OctoPrint-ExtraFileInfo @ file:///tmp/tmp8ch4y05c/OctoPrint-ExtraFileInfo-master.zip
OctoPrint-FileCheck==2021.2.23
OctoPrint-FileManager @ file:///tmp/tmppe9jiqxg/OctoPrint-FileManager-master.zip
OctoPrint-FirmwareCheck==2021.10.11
OctoPrint-HeaterTimeout @ file:///tmp/tmpt919dljq/OctoPrint-HeaterTimeout-master.zip
OctoPrint-HomeAssistant @ file:///tmp/tmp8q3xc4xm/OctoPrint-HomeAssistant-master.zip
OctoPrint-Marlin-Flasher @ file:///tmp/tmpizkkg13l/OctoPrint-Marlin-Flasher-master.zip
OctoPrint-MeatPack @ file:///tmp/tmp6dh4vae5/OctoPrint-MeatPack-master.zip
OctoPrint-MQTT @ file:///tmp/tmpwz9mbvt0/OctoPrint-MQTT-master.zip
OctoPrint-MQTTPublish @ file:///tmp/tmpfe1hq37g/OctoPrint-MQTTPublish-master.zip
OctoPrint-MQTTSubscribe @ file:///tmp/tmp2npyyzmy/OctoPrint-MQTTSubscribe-master.zip
OctoPrint-PiSupport==2023.7.31
OctoPrint-PSUControl @ file:///tmp/tmpopbv4qae/OctoPrint-PSUControl-master.zip
OctoPrint-PSUControl-HomeAssistant @ file:///tmp/tmpj981kxom/OctoPrint-PSUControl-HomeAssistant-main.zip
OctoPrint-Resource-Monitor @ file:///tmp/tmp08v7dsi_/OctoPrint-Resource-Monitor-master.zip
OctoPrint-SlicerSettingsParser @ file:///tmp/tmppsefvatn/OctoPrint-SlicerSettingsParser-master.zip
OctoPrint-SmartPreheat @ file:///tmp/tmpazygkl2y/OctoPrint-SmartPreheat-master.zip
OctoPrint-TabOrder @ file:///tmp/tmpc5lq9kqp/OctoPrint-TabOrder-master.zip
OctoPrint-WideScreen @ file:///tmp/tmpa79q1ozb/OctoPrint-WideScreen-master.zip
OctoPrint-WLED @ file:///tmp/tmpfucxog4z/release.zip
ordered-set==4.1.0
packaging==23.1
paho-mqtt==1.6.1
passlib==1.7.4
pathvalidate==2.5.2
Pillow==10.0.1
pkginfo==1.9.6
platformdirs==3.10.0
ply==3.11
Preheat @ file:///tmp/tmph_d_3gb6/octoprint-preheat-master.zip
psutil==5.9.5
pycparser==2.21
pydantic==1.10.12
pyduinocli==0.31.0
Pygments==2.16.1
pylru==1.2.1
pyserial==3.5
python-dateutil==2.8.2
python-periphery==2.4.1
pytz==2023.3.post1
PyYAML==6.0.1
regex==2023.8.8
requests==2.31.0
rich==13.5.3
RPi.GPIO==0.7.1
sarge==0.1.7.post1
semantic-version==2.10.0
sentry-sdk==1.31.0
sgmllib3k==1.0.0
six==1.16.0
Slicer-Thumbnails @ https://github.com/jneilliii/OctoPrint-PrusaSlicerThumbnails/archive/1.0.3.zip
Themeify @ file:///tmp/tmptqwo9rbe/OctoPrint-Themeify-master.zip
tornado==6.3.3
typing_extensions==4.8.0
Unidecode==1.3.7
Upload-Anything @ file:///tmp/tmpldpd95u6/UploadAnything-master.zip
urllib3==2.0.5
Virtual-Printer-Settings @ file:///tmp/tmp7ij0_kfm/release.zip
virtualenv==20.24.5
watchdog==2.3.1
webassets==2.0
WebDAV-Backup @ https://github.com/edekeijzer/OctoPrint-WebDavBackup/archive/1.0.0.zip
webdavclient3==3.14.6
websocket-client==1.6.3
Werkzeug==2.2.3
wrapt==1.15.0
yarl==1.9.2
zeroconf==0.39.4
zipstream-ng==1.6.0
Oh wow, that's a number of plugins... can you reproduce the same issue in a clean venv of the same python version with just OctoPrint 1.9.3 installed and pydantic downgraded to 1.10.12?
I can't imagine that a plugin is at fault here, but it would still be a helpful data point and possibly allow me to narrow things down.
@edekeijzer forgot the ping, see above
I'm also running it in a venv on a Raspberry Pi 3 and Raspbian Bookworm.
Both versions 1.9.2 and 1.9.2 are throwing the same error on startup.
After downgrading pydantic==1.10.10 both versions are starting without errors.
I can reproduce this in an arm32v7 bookworm docker container with RPi packages and Python 3.11.2 but not on Python 3.11.4 on my local machine or Python 3.11.2 on the CI. And also not against Python 3.9 on bullseye on RPi.
So I got curious and reconfigured the docker container to NOT use the RPi packages, but same architecture and still Python 3.11.2, and it turns out I can no longer repro.
So this is apparently currently isolated to RaspberryPi OS Bookworm Python 3.11.2. And on that, I need to downgrade pydantic to 1.10.10 to make things work again.
@foosel having a busy day at work, but I just managed to test with a clean venv - same result. However, downgrading to octoprint 1.9.2 and pydantic 1.10.10 got my installation back to work. Thanks for the workaround, please keep me posted if you find anything.
At least I now got something that allows a bit easier turnaround while testing, octoprint config get . not only triggers the error but gives us a nice stack trace as well:
(octoprint) root@19debc6ace2a:/octoprint# octoprint config get .
Traceback (most recent call last):
File "/octoprint/bin/octoprint", line 8, in <module>
sys.exit(main())
^^^^^^
File "/octoprint/lib/python3.11/site-packages/octoprint/__init__.py", line 1008, in main
octo(args=args, prog_name="octoprint", auto_envvar_prefix="OCTOPRINT")
File "/octoprint/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/octoprint/lib/python3.11/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/octoprint/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/octoprint/lib/python3.11/site-packages/octoprint/cli/common.py", line 37, in invoke
return self._impl.invoke(ctx)
^^^^^^^^^^^^^^^^^^^^^^
File "/octoprint/lib/python3.11/site-packages/click/core.py", line 1685, in invoke
super().invoke(ctx)
File "/octoprint/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/octoprint/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/octoprint/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context(), *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/octoprint/lib/python3.11/site-packages/octoprint/cli/config.py", line 57, in cli
ctx.obj.settings = init_settings(
^^^^^^^^^^^^^^
File "/octoprint/lib/python3.11/site-packages/octoprint/__init__.py", line 203, in init_settings
from octoprint.settings import InvalidSettings, settings
File "/octoprint/lib/python3.11/site-packages/octoprint/settings/__init__.py", line 35, in <module>
from octoprint.schema.config import Config
File "/octoprint/lib/python3.11/site-packages/octoprint/schema/config/__init__.py", line 9, in <module>
from .access_control import AccessControlConfig
File "/octoprint/lib/python3.11/site-packages/octoprint/schema/config/access_control.py", line 11, in <module>
class AccessControlConfig(BaseModel):
File "pydantic/main.py", line 186, in pydantic.main.ModelMetaclass.__new__
TypeError: Argument 'bases' has incorrect type (expected list, got tuple)
Ok, so the issue at least on Bookworm with Python 3.11 on armv7 (so in short, any 32bit Pi running RPi OS Bookworm) with pydantic stems from the binary wheels available on piwheels! Those are broken for pydantic 1.10.[11,12,13]. 1.10.10 isn't available on piwheels, so if you try to install that it will be fetched from pypi and work. I've documented my findings in pydantic/pydantic#7689 (comment) and suggested to pydantic to request a takedown of the broken binary wheels from piwheels. If they do that I won't have to roll out another bugfix release, if they don't I'll look into pushing a pin to 1.10.10 early next week.
edit I've also reported this on the piwheels issue tracker as piwheels/packages#396
Nothing to do here, that's all up to pydantic and piwheels.