Error with cookies
daniarla opened this issue · comments
Avoid duplicates
- Bug is not already reported in another issue
Describe the bug
Every 48h or so for the last week I started to get errors like this when running moodle-dl.
File "/usr/lib64/python3.11/http/cookiejar.py", line 2041, in _really_load
domain, domain_specified, path, secure, expires, name, value = \
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: not enough values to unpack (expected 7, got 1)
_warn_unhandled_exception()
The following error occurred during execution:
invalid Netscape format cookies file 'Cookies.txt': 'xxxxxxxx'
Then I remove the cookies.txt file manually and generate a new token with --init --new-token, it fails a couple of times getting the auto-login key with Failed to download autologin key! and later it works for a day without problems.
`
Steps to reproduce the issue
Just running moodle-dl as always.
Technical details
- OS: Fedora 37
- Version 2.2.2.3
I'm using sso logging in.
Its seems a problem with the library because for some reason the file sometimes looks like this:
# Netscape HTTP Cookie File
# http://curl.haxx.se/rfc/cookie_spec.html
# This is a generated file! Do not edit.
moodleinstance.es FALSE / TRUE MoodleSession xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxx
Where the x is the cookie. It seems like sometimes there is extra text in the file and generates an error.
If your moodle does not have any plugins that need cookies to download, you could also just disable downloading with cookie in the settings and remove the private token from the config.json. This does not solve the problem, but probably will mitigate the symptom.
For an better understanding what is happening, you shluld send me the cookies.txt when the error happens again (e.g. in Discord). Also you do not need to relogin every time you remove the cookies.txt. This file is only extra and is not required for the main functionallity. It is regenerated automaticly on the next run (if you have set an private token in your config).
Sorry for the late reply! Whats you discord username? It just happened again so I have logged the error and I'll send it asap.
Also, I think that the problem is related to regenerating when there is already a cookies.txt present. Deleting the file (or in this case changing the name of the file so I can send it to you) and running moodle-dl again downloads a new and valid cookie without a problem.
Sorry for the delay. My username is c0d3d3v#8812 you can also find me on the server that is linked in the Readme.
It will be hard to track down the issue, but maybe the broken cookies.txt gives me an hint.
Just a heads up! I sent you the info via discord.
Hi @daniarla,
In the last few days moodle-dl stopped working for me also with the same error as yours.
The only change I made was to update my main python version to 3.11 (the same version you are using).
I can confirm that deleting Cookies.txt and running Moodle-dl under python 3.9 seems to work (but I will look for it for the next two days), so we can use it for now as a workaround.
My guess is that this happens because either multiple instances of yt-dlp or multiple threads of moodle-dl are trying to write to the cookie.txt file at the same time. I will think of something to make writing to the file more secure. Maybe we just give yt-dlp a copy and ignore all changes from yt-dlp to the file.
Maybe related to yt-dlp/yt-dlp#5977
I'm correcting myself, moodle-dl does only write to the cookie file once at the startup and does not write again in the download threads. So it is yt-dlp that is overwriting the file. I will implement a mechanic to give yt-dlp a temporary cookie file and then merge it maybe back to the original file or just delete it (probably to fix the issue we just delete the file, since keeping the cookies up to date is not a main feature of moodle-dl, but maybe someday we will implement it).
I will maybe ask on the yt-dlp repo for an option to not write back to the cookie file, that would also solve the problem.
Mh, I'm not sure if this is really the problem. Since we actually do not open YT-DLP in a context (like yt-dlp itself does https://github.com/yt-dlp/yt-dlp/blob/203a06f8554df6db07d8f20f465ecbfe8a14e591/yt_dlp/__init__.py#L930) and so the the cookiejar should not be saved since exit gets not called https://github.com/yt-dlp/yt-dlp/blob/203a06f8554df6db07d8f20f465ecbfe8a14e591/yt_dlp/YoutubeDL.py#L915-L919
One more guess would be that it is the same issue that got fixed years ago in yt-dlp (see yt-dlp/yt-dlp@1d88b3e and yt-dlp/yt-dlp@1bab343) that somehow the expires field of the cookie is None and so gets stored as an empty string. I think I will copy the behaviour of YoutubeDLCookieJar.
As soon as I implemented it I will ping you for testing :)
Fix is released