"Fake Acknowledgement" required to get past bad serial parses?
nbergen opened this issue · comments
The problem
My setup has been running without issue for a bit more than a year. Recently, Octoprint was inadvertently updated from 1.8.3 to 1.9.3, and I began having issues with prints.
Description
The issue manifests by hanging the print until the communication timeout expires. This increases the time to print as well as ruining the print with blobbing as the printhead remains motionless for a few seconds.
It can be partially, though insufficiently mitigated by both reducing the printer communication timeout, and running a modified version of the old octoprint_ackackack plugin:
https://gist.github.com/nbergen/abb07a55df30b05f6aa8ce6e2d771aa3
It appears to have to do with serial parsing, as sometimes concatenated lines can plainly be seen.
Other times, there appears to be no OK in the stream. This commonly happens immediately after starting the print, it transitions to "Starting", but will not set the target temperature until "Fake Acknowledgement" is pressed (or the case is manually handled in octoprint_ackackack).
Similar issues
I've found a handful of similar issues of course, both in Marlin and Octoprint, but none appear to match exactly with what I'm seeing.
- https://community.octoprint.org/t/serial-connection-commands-bufferend-until-fake-ack/11839/7
- #2454
- #2380
- MarlinFirmware/Marlin#26507
- https://www.reddit.com/r/VoxelabAquila/comments/rwoovl/h32_blobbing_with_only_octoprint_or_all_usb/
Caused by the upgrade?
I do see a note in #2454 that this is suspected to be an issue in the underlying USB/Serial processing libraries, though I was fine before the upgrade to 1.9.3. The issue now reproduces in all Octoprint versions I've tested from 1.8.2 through 1.10.3rc, which makes me wonder if some other supporting library was upgraded with the Octoprint upgrade that's causing the issue.
Firmware
I am running an old version of Marlin's bugfix branch, and I haven't ventured into troubleshooting the firmware, though I have read the notes on increasing the buffer size, as
- I was stable with this firmware for more than a year.
- I no longer have the dev environment set up to build the firmware.
- It was rather finicky to find a config the printer was happy with.
Did the issue persist even in safe mode?
Yes, it did persist
If you could not test in safe mode, please state why ("currently printing" is NOT an excuse!)
No response
Version of OctoPrint
1.9.3
Operating system running OctoPrint
Ubuntu 20.04.5 LTS
Proxmox VM -- should be plenty of CPU.
Printer model & used firmware incl. version
Ender5: MarlinFirmware/Marlin:bugfix-2.1.x@5809bdd (old marlin bugfix branch)
SKR mini board & LCD
Browser and version of browser, operating system running browser
No response
Checklist of files to include below
- Systeminfo Bundle (always include!)
- Contents of the JavaScript browser console (always include in cases of issues with the user interface)
- Screenshots and/or videos showing the problem (always include in case of issues with the user interface)
- GCODE file with which to reproduce (always include in case of issues with GCODE analysis or printing behaviour)
Additional information & file uploads
octoprint-systeminfo-20240211003037.zip
Console log
Less has finished and no sheets were loaded.
packed_core.js?d7656e66:1755 Starting dependency resolution...
packed_core.js?d7656e66:1768 ... dependency resolution done
packed_core.js?d7656e66:1791 Initial application setup done, connecting to server...
packed_core.js?d7656e66:1585 Connected to the server
packed_core.js?d7656e66:1586 ❗ Safe mode is active. Third party plugins and language packs are disabled and cannot be enabled.
packed_core.js?d7656e66:1586 ❗ Reason for safe mode: settings
packed_core.js?d7656e66:1590 Triggering connect on all view models
packed_core.js?d7656e66:1592 Server (re)connect processed
packed_core.js?d7656e66:1791 Finalizing application startup
packed_core.js?d7656e66:1772 Going to bind 38 view models...
packed_core.js?d7656e66:1784 Did not bind view model BackupViewModel to target #wizard_plugin_backup since it does not exist
packed_core.js?d7656e66:1784 Did not bind view model ClassicWebcamWizardViewModel to target #wizard_classicwebcam since it does not exist
packed_core.js?d7656e66:823 Chrome Arc Fix disabled
packed_core.js?d7656e66:823 Chrome Arc Fix enabled
packed_core.js?d7656e66:1784 Did not bind view model SoftwareUpdateViewModel to target #softwareupdate_confirmation_dialog since it does not exist
packed_core.js?d7656e66:1784 Did not bind view model SoftwareUpdateViewModel to target #wizard_plugin_softwareupdate_update since it does not exist
packed_core.js?d7656e66:1784 Did not bind view model SoftwareUpdateViewModel to target #wizard_plugin_softwareupdate_settings since it does not exist
packed_core.js?d7656e66:235 User octoprint logged in
packed_core.js?d7656e66:240 Login mechanism: remember_me
packed_core.js?d7656e66:1789 ... binding done
packed_core.js?d7656e66:1789 Application startup complete
Proceeds on Fake Acknowledgement
Concatenated 'ok'
(probably partial) list of bad OKs in safemode
I've also seen them smack in the middle of 2 temperature infos
cat serial.log | grep "[a-zA-Z0-9]ok"
2024-02-11 00:35:36,096 - Recv: T:210.21 /210.00 B:64.84 /65.00ok
2024-02-11 00:37:06,176 - Recv: T:209.87 /210.00 B:60.08 /60.00ok
2024-02-11 00:37:56,111 - Recv: T:210.07 /210.00 B:60.61 /60.00ok
2024-02-11 00:38:26,124 - Recv: T:209.41 /210.00 B:59.40 /60.00ok
2024-02-11 00:38:46,199 - Recv: T:209.98 /210.00 B:60.52 /60.00ok
2024-02-11 00:41:30,163 - Recv: T:209.93 /210.00 B:59.64 /60.00ok
2024-02-11 00:42:06,230 - Recv: T:209.88 /210.00 B:60.25 /60.00ok
2024-02-11 00:42:42,247 - Recv: T:209.56 /210.00 B:59.88 /60.00ok
2024-02-11 00:43:22,258 - Recv: T:209.47 /210.00 B:59.82 /60.00ok
2024-02-11 00:44:18,203 - Recv: T:210.34 /210.00 B:59.99 /60.00ok
2024-02-11 00:44:38,195 - Recv: T:210.34 /210.00 B:60.00 /60.00ok
2024-02-11 00:45:12,293 - Recv: T:209.38 /210.00 B:60.05 /60.00ok
2024-02-11 00:46:36,357 - Recv: T:209.53 /210.00 B:59.97 /60.00ok
2024-02-11 00:47:50,368 - Recv: T:210.07 /210.00 B:60.10 /60.00ok
2024-02-11 00:48:32,260 - Recv: T:210.02 /210.00 B:60.11 /60.00ok
2024-02-11 00:49:48,453 - Recv: T:209.89 /210.00 B:59.86 /60.00ok
2024-02-11 00:50:22,543 - Recv: T:209.91 /210.00 B:60.06 /60.00ok
2024-02-11 00:50:42,381 - Recv: T:210.61 /210.00 B:59.84 /60.00ok
2024-02-11 00:51:12,369 - Recv: T:209.92 /210.00 B:60.11 /60.00ok
2024-02-11 00:51:32,436 - Recv: T:210.29 /210.00 B:59.88 /60.00ok
2024-02-11 00:52:06,489 - Recv: T:209.65 /210.00 B:60.14 /60.00ok
2024-02-11 00:52:30,272 - Recv: T:210.21 /210.00 B:59.79 /60.00ok
2024-02-11 00:52:50,327 - Recv: T:210.25 /210.00 B:60.20 /60.00ok
2024-02-11 00:53:30,513 - Recv: T:209.91 /210.00 B:59.82 /60.00ok
2024-02-11 00:54:06,416 - Recv: T:209.74 /210.00 B:60.00 /60.00ok
2024-02-11 00:54:28,384 - Recv: T:210.26 /210.00 B:59.87 /60.00ok
2024-02-11 00:55:00,287 - Recv: T:209.56 /210.00 B:60.10 /60.00ok
2024-02-11 00:56:42,413 - Recv: T:209.96 /210.00 B:59.98 /60.00ok
2024-02-11 00:57:02,329 - Recv: T:210.15 /210.00 B:59.98 /60.00ok
2024-02-11 00:57:22,451 - Recv: T:209.95 /210.00 B:60.01 /60.00ok
2024-02-11 00:59:28,376 - Recv: T:209.83 /210.00 B:59.84 /60.00ok
2024-02-11 00:59:54,445 - Recv: T:210.25 /210.00 B:60.20 /60.00ok
2024-02-11 01:00:48,463 - Recv: T:209.91 /210.00 B:60.29 /60.00ok
2024-02-11 01:01:06,414 - Recv: T:210.50 /210.00 B:59.99 /60.00ok
2024-02-11 01:01:26,360 - Recv: T:210.16 /210.00 B:59.92 /60.00ok
2024-02-11 01:01:52,851 - Recv: T:210.10 /210.00 B:60.02 /60.00ok
2024-02-11 01:04:18,508 - Recv: T:210.00 /210.00 B:59.73 /60.00ok
2024-02-11 01:05:44,376 - Recv: T:210.02 /210.00ok
2024-02-11 01:07:06,418 - Recv: T:209.89 /210.00 B:60.15 /60.00ok
2024-02-11 01:07:24,390 - Recv: T:209.85 /210.00 B:59.88 /60.00ok
>$ cat serial.log | grep "ok[a-zA-Z0-9]"
>$ cat serial.log | grep "ok[a-zA-Z 0-9]"
2024-02-11 00:23:48,950 - Recv: ok T:36.22 /0.00 B:39.37 /65.00 @:0 B@:0
2024-02-11 00:25:40,899 - Recv: ok T:33.11 /210.00 B:65.83 /65.00 T:33.09 /210.00 B:65.97 /65.00 @:127 B@:73
2024-02-11 00:35:52,104 - Recv: ok T:210.33 /210.00 B:64.57 /65.00 @:66 B@:82
2024-02-11 00:37:22,132 - Recv: ok T:210.14 /210.00 @:74 B@:72
2024-02-11 00:37:48,119 - Recv: ok T:209.94 /210.00 B:60.68 /60.00 @:79 B@:40
2024-02-11 00:38:12,127 - Recv: ok T:210.14 /210.00 B:59.74 /60.00 @:76 B@:28
2024-02-11 00:38:42,129 - Recv: ok T:210.40 /210.00 B:60.25 /60.00 @:73 B@:61
2024-02-11 00:39:02,135 - Recv: ok T:210.01 /210.00 B:60.28 /60.00 @:79 B@:20
2024-02-11 00:40:32,160 - Recv: ok T:210.10 /210.00 B:59.70 /60.00 @:79 B@:68
2024-02-11 00:41:12,177 - Recv: ok T:210.21 /210.00 B:59.96 /60.00 @:78 B@:33
2024-02-11 00:41:46,186 - Recv: ok T:210.42 /210.00 B:60.08 /60.00 @:74 B@:61
2024-02-11 00:42:22,186 - Recv: ok T:210.36 /210.00 B:59.87 /60.00 @:75 B@:38
2024-02-11 00:42:58,190 - Recv: ok T:210.31 /210.00 B:60.28 /60.00 @:77 B@:43
2024-02-11 00:43:38,195 - Recv: ok T:210.40 /210.00 B:59.97 /60.00 @:74 B@:50
2024-02-11 00:44:34,210 - Recv: ok T:210.04 /210.00 B:59.92 /60.00 @:80 B@:49
2024-02-11 00:44:54,209 - Recv: ok T:210.09 /210.00 B:60.08 /60.00 @:77 B@:46
2024-02-11 00:45:28,215 - Recv: ok T:210.25 /210.00 B:60.07 /60.00 @:77 B@:40
2024-02-11 00:46:22,222 - Recv: ok T:210.62 /210.00 B:59.93 /60.00 @:69 B@:49
2024-02-11 00:46:52,218 - Recv: ok T:210.40 /210.00 B:59.96 /60.00 @:72 B@:50
2024-02-11 00:48:08,230 - Recv: ok T:210.55 /210.00 B:59.90 /60.00 @:71 B@:53
2024-02-11 00:48:48,247 - Recv: ok T:210.32 /210.00 B:59.97 /60.00 @:73 B@:50
2024-02-11 00:50:04,242 - Recv: ok T:210.33 /210.00 B:60.17 /60.00 @:73 B@:47
2024-02-11 00:50:38,235 - Recv: ok T:210.23 /210.000.00 @:74 B@:48
2024-02-11 00:50:58,246 - Recv: ok T:209.78 /210.00 @:80 B@:45
2024-02-11 00:51:28,262 - Recv: ok T:209.92 /210.00 B:59.79 /60.00 @:81 B@:54
2024-02-11 00:51:48,266 - Recv: ok T:210.16 /210.00 B:60.05 /60.00 @:75 B@:50
2024-02-11 00:52:22,279 - Recv: ok T:210.05 /210.00 B:59.74 /60.00 @:78 B@:50
2024-02-11 00:52:46,271 - Recv: ok T:210.14 /210.00 B:60.14 /60.00 @:76 B@:50
2024-02-11 00:53:06,290 - Recv: ok T:210.35 /210.00 B:60.21 /60.00 @:70 B@:29
2024-02-11 00:53:46,284 - Recv: ok T:210.16 /210.00 B:60.01 /60.00 @:78 B@:49
2024-02-11 00:54:22,293 - Recv: ok T:209.75 /210.00 B:59.86 /60.00 @:86 B@:47
2024-02-11 00:54:44,291 - Recv: ok T:209.88 /210.00 B:59.98 /60.00 @:81 B@:51
2024-02-11 00:56:58,307 - Recv: ok T:210.24 /210.00 B:59.96 /60.00 @:76 B@:48
2024-02-11 00:57:18,307 - Recv: ok T:210.10 /210.00 B:60.02 /60.00 @:76 B@:46
2024-02-11 00:57:38,316 - Recv: ok T:210.04 /210.00 B:59.97 /60.00 @:77 B@:47
2024-02-11 00:59:44,331 - Recv: ok T:209.94 /210.00 B:60.14 /60.00 @:81 B@:45
2024-02-11 01:00:10,337 - Recv: ok T:210.44 /210.00 @:71 B@:38
2024-02-11 01:01:04,341 - Recv: ok T:210.36 /210.00 B:60.08 /60.00 @:74 B@:36
2024-02-11 01:01:22,349 - Recv: ok T:209.92 /210.00 B:59.84 /60.00 @:77 B@:48
2024-02-11 01:01:42,355 - Recv: ok T:209.74 /210.00 B:60.03 /60.00 @:81 B@:46
2024-02-11 01:02:08,358 - Recv: ok T:210.36 /210.00 B:59.83 /60.00 @:71 B@:52
2024-02-11 01:03:58,365 - Recv: ok T:209.96 /210.00 B:60.19 /60.00 @:82 B@:28
2024-02-11 01:04:34,374 - Recv: ok T:209.81 /210.00 B:59.99 /60.00 @:83 B@:58
2024-02-11 01:04:58,405 - Recv: ok T:209.86 /210.00 B:60.18 /60.00 @:83 B@:30
2024-02-11 01:06:00,381 - Recv: ok T:210.09 /210.00 B:60.21 /60.00 @:78 B@:34
2024-02-11 01:07:22,390 - Recv: ok T:210.02 /210.00 B:59.94 /60.00 @:77 B@:41
2024-02-11 01:07:40,404 - Recv: ok T:209.65 /210.00 B:59.93 /60.00 @:86 B@:53
pip list
> pip list
Package Version
----------------------------- -----------
anyio 3.6.1
argon2-cffi 21.3.0
argon2-cffi-bindings 21.2.0
async-timeout 4.0.3
asyncclick 8.1.3.4
Babel 2.12.1
backports.csv 1.0.7
Bed-Visualizer 1.1.1
blinker 1.7.0
cachelib 0.10.2
certifi 2024.2.2
cffi 1.16.0
charset-normalizer 3.3.2
class-doc 0.2.6
click 8.1.7
colorlog 6.8.2
commonmark 0.9.1
Deprecated 1.2.14
emoji 2.10.1
feedparser 6.0.11
filetype 1.2.0
Flask 2.2.5
Flask-Assets 2.1.0
flask-babel 3.1.0
Flask-Limiter 3.5.0
Flask-Login 0.6.3
frozendict 2.4.0
future 0.18.3
Heatshrink 0.3.2
heatshrink2 0.11.0
idna 3.6
ifaddr 0.2.0
importlib-metadata 7.0.1
importlib-resources 6.1.1
itsdangerous 2.1.2
Jinja2 3.1.3
limits 3.7.0
Markdown 3.5.2
markdown-it-py 3.0.0
MarkupSafe 2.1.5
marlin-binary-protocol 0.0.7
Marlin-GCode-Documentation 0.18.0
mdurl 0.1.2
more-itertools 10.2.0
netaddr 0.8.0
netifaces 0.11.0
netifaces2 0.0.21
numpy 1.23.3
OctoPrint 1.9.3
OctoPrint-AutoTerminalInput 0.1.0
OctoPrint-Cancelobject 0.4.7
OctoPrint-CrealityTemperature 1.2.4
OctoPrint-Dashboard 1.19.10
OctoPrint-Display-ETA 1.0.2
OctoPrint-ExcludeRegionPlugin 0.3.2
OctoPrint-FilamentManager 1.9.1
OctoPrint-FileCheck 2021.2.23
OctoPrint-FirmwareCheck 2021.10.11
OctoPrint-FirmwareUpdater 1.14.0
OctoPrint-Fullscreen 0.0.6
OctoPrint-HeaterTimeout 0.0.4
OctoPrint-NavbarTemp 0.15
OctoPrint-OctoPod 0.3.16
OctoPrint-PiSupport 2023.10.10
OctoPrint-PSUControl 1.0.6
OctoPrint-PSUControl-TPLink 1.0.1
OctoPrint-Tplinkautoshutdown 1.3.0
OctoPrint-TPLinkSmartplug 1.0.3
OctoPrint-WebcamTab 0.3.1
ordered-set 4.1.0
packaging 23.2
pandas 1.5.0
passlib 1.7.4
pathvalidate 2.5.2
peewee 3.15.3
Pillow 9.2.0
pip 24.0
pkg_resources 0.0.0
pkginfo 1.9.6
psutil 5.9.8
pycparser 2.21
pydantic 1.10.12
Pygments 2.17.2
pylru 1.2.1
pyparsing 3.0.9
pyserial 3.5
python-dateutil 2.8.2
python-kasa 0.5.0
python-periphery 2.4.1
pytz 2024.1
PyYAML 6.0.1
qrcode 7.3.1
regex 2023.12.25
requests 2.31.0
rich 13.7.0
sarge 0.1.7.post1
semantic-version 2.10.0
sentry-sdk 1.40.3
setuptools 44.0.0
sgmllib3k 1.0.0
Sidebar-Webcam 0.1.8
six 1.16.0
sniffio 1.3.0
SQLAlchemy 1.1.18
Terminal-Commands-Extended 0.1.7
tinydb 4.7.0
tornado 6.4
typing_extensions 4.9.0
Unidecode 1.3.8
uptime 3.0.1
uritools 2.1.1
urllib3 2.2.0
watchdog 2.3.1
webassets 2.0
websocket-client 1.7.0
Werkzeug 2.2.3
wheel 0.42.0
wrapt 1.15.0
zeroconf 0.39.4
zipp 3.17.0
zipstream-ng 1.7.1
Your softwareupdate log should contain everything that got updated along side OctoPrint with regards to Python packages, so that's where I'd start to dig. Unfortunately the one contained in your bundle only contains a plugin update from Feb 10th. The update log shows only one update of OctoPrint from 1.9.2 to 1.10.0rc1, no information seems to be available about your mentioned update from 1.8.3 to 1.9.3.
Can you please verify that you've shared the bundle of the correct instance? Do you maybe still have the software update log that contains the update you went through from 1.8.3 to 1.9.3?
What I can see in the commit log is that I upgraded the dependency on pyserial from 3.4 to 3.5 between 1.8.x and 1.9.0. What you could try is rolling back to pyserial 3.4 (pip install pyserial==3.4 - this will probably log a big warning about unmet dependencies then since OctoPrint states it expects 3.5, but it should still work I hope) and see if that improves things again.
If it is indeed the pyserial update that triggered that (and that's tbh the only thing I could imagine having an impact here), it would match your observation that downgrading OctoPrint didn't solve it, as that wouldn't downgrade pyserial alongside it as long as its current version is within the stated version range, which 3.5 indeed is even for OctoPrint 1.8.3.
Oh, also, do you happen to have "Request low latency mode" in Settings > Serial Connection > Connection?