Fails with recent curl (e.g., macOS 10.14.6+, 10.15+, a.o.) "certificate has expired"
cox890 opened this issue · comments
The download & return suddenly stopped working I have tried with a fresh install same problem a .odm.license is created by the script it's empty & nothing happens
same here - same error. Tried with both the current and an older version.
What OS are you all using? It still works for me on Debian.
OSX - Mojave 10.14.6 - been using this for a couple of years but the problem only appeared a week or so ago.
I tried this on Amazon Linux 2 and have the same problem. but I was able to run it on CentOS 8.4 (with a fresh ODM in each case)
I also have this problem and am also using OSX Mojave 10.14.6. The output does this:
ClientID=
Using AcquisitionUrl=
Using MediaID=
Using RawHash=
Using Hash=
but nothing downloads.
Have you tried using the --verbose flag to see what might be going on? Also, which version of xmllint do you have installed?
This on my Mac (Mojave):
I edited overdrive.sh and added '-v' to the curl options and it reports a certificate expiration error
I added -k (ignore ssl errors) to the curl opts (getting a fresh ODM) and sure enough it works - I'm donloading files
so I directly check the certificate:
$ openssl s_client -connect 207.54.136.40:443 | openssl x509 -noout -text
depth=1 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMT
verify return:0
depth=1 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMT
verify return:0
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMT
verify return:0
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
03:81:e3:4c:c8:6a:7b:e6:ca:93:d2:a0:1b:e9:d6:01:dc:f6
Signature Algorithm: sha256WithRSAEncryption
Issuer: C=US, O=Let's Encrypt, CN=R3
Validity
Not Before: Aug 9 16:20:39 2021 GMT
Not After : Nov 7 16:20:37 2021 GMT
Subject: CN=*.cdn.overdrive.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
...etc....
The certs expired Sept 30 - which right when I started having problems.
It looks like they've switched to Lets Encrypt but never removed the expired certs. Now I'm not sure why some are choking and others are fine with this - maybe different versions of curl? my Mac is running 7.54.0 while my cent box is 7.61
As a workaround I think I'm not seeing too much risk in using -k on curl...
29c29
< CURLOPTS=(-s -L -A "$UserAgent" --compressed)
CURLOPTS=(-s -k -L -A "$UserAgent" --compressed)
Not OP but here's my output. overdrive download *.odm --verbose
Entering debug (verbose) mode
+ CURLOPTS=("${CURLOPTS[@]:1}")
+ shift
+ [[ 0 -gt 0 ]]
+ [[ 1 -eq 0 ]]
+ [[ 1 -eq 0 ]]
+ HEADER_PRINTED=
+ for ODM in '"${MEDIA[@]}"'
+ for COMMAND in '"${COMMANDS[@]}"'
+ case $COMMAND in
+ download BellJar-7138.odm
+ license_path=BellJar-7138.odm.license
+ acquire_license BellJar-7138.odm BellJar-7138.odm.license
+ [[ -e BellJar-7138.odm.license ]]
++ uuidgen
++ tr '[:lower:]' '[:upper:]'
+ ClientID=ID
+ printf 'Generating random ClientID=%s\n' DD2537FF-5F56-4375-B515-02BA63949318
Generating random ClientID=DD2537FF-5F56-4375-B515-02BA63949318
++ xmllint --xpath '/OverDriveMedia/License/AcquisitionUrl/text()' BellJar-7138.odm
+ AcquisitionUrl=https://ofs.contentreserve.com/MP3LicenseAcquisitionService.svc/ab3d7e4e-90c5-48cc-93f3-adb60078a400
+ printf 'Using AcquisitionUrl=%s\n' https://ofs.contentreserve.com/MP3LicenseAcquisitionService.svc/ab3d7e4e-90c5-48cc-93f3-adb60078a400
Using AcquisitionUrl=https://ofs.contentreserve.com/MP3LicenseAcquisitionService.svc/ab3d7e4e-90c5-48cc-93f3-adb60078a400
++ xmllint --xpath 'string(/OverDriveMedia/@id)' BellJar-7138.odm
+ MediaID=348c3c61-b9d4-402e-873f-b2d10f610395-425
+ printf 'Using MediaID=%s\n' 348c3c61-b9d4-402e-873f-b2d10f610395-425
Using MediaID=348c3c61-b9d4-402e-873f-b2d10f610395-425
+ RawHash='DD2537FF-5F56-4375-B515-02BA63949318|1.2.0|10.11.6|ELOSNOC*AIDEM*EVIRDREVO'
+ printf 'Using RawHash=%s\n' 'DD2537FF-5F56-4375-B515-02BA63949318|1.2.0|10.11.6|ELOSNOC*AIDEM*EVIRDREVO'
Using RawHash=DD2537FF-5F56-4375-B515-02BA63949318|1.2.0|10.11.6|ELOSNOC*AIDEM*EVIRDREVO
++ echo -n 'DD2537FF-5F56-4375-B515-02BA63949318|1.2.0|10.11.6|ELOSNOC*AIDEM*EVIRDREVO'
++ iconv -f ASCII -t UTF-16LE
++ openssl dgst -binary -sha1
++ base64
+ Hash=K9n1Of/bjq/1tNMC9cxjDvuCt5A=
+ printf 'Using Hash=%s\n' K9n1Of/bjq/1tNMC9cxjDvuCt5A=
Using Hash=K9n1Of/bjq/1tNMC9cxjDvuCt5A=
+ curl -L -A 'OverDrive Media Console' --compressed 'https://ofs.contentreserve.com/MP3LicenseAcquisitionService.svc/ab3d7e4e-90c5-48cc-93f3-adb60078a400?MediaID=348c3c61-b9d4-402e-873f-b2d10f610395-425&ClientID=ID=1.2.0&OS=10.11.6&Hash=K9n1Of/bjq/1tNMC9cxjDvuCt5A='
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
HTTPS-proxy has similar options --proxy-cacert and --proxy-insecure
I just pulled a newer version of curl from brew, 7.79.1, and removed the -k, and downloaded another book just fine.
Definitely relating to the expired certificates and how your version of CURL deals with it.
If you do get the brew version, realize brew doesn't replace the OSX version in /usr/bin/curl so you will need to use curl from /usr/local/opt/curl/bin
I did this by adding:
export PATH="/usr/local/opt/curl/bin:$PATH"
to overdirve.sh (the -k was removed for my test.)
Thanks for your help Glocktober! It worked after I updated curl from brew running the extra PATHs you and brew itself suggest. I didn't remove the -k in overdrive.sh.
First off just wanna say I'm chuffed that you all are helping diagnose and debug before I even I get a chance to catch up on my email and poke my head in :)
As @Glocktober mentioned, there's some (temporary?) issue with the SSL certs; not sure what changed in curl, or why macOS's system curl has a problem with them — they look ok as far as SSL Labs is concerned — but tl;dr:
-CURLOPTS=(-s -L -A "$UserAgent" --compressed)
+CURLOPTS=(-s -L -A "$UserAgent" --compressed --insecure)seems like an entirely reasonable fix; it's not like you're sending any sensitive info or going to be devastated if someone actually MITM's your connection and Rickrolls your audiobook ;)
From the output of:
openssl s_client -connect mp3audio-gk.cdn.overdrive.com:443
it looks like overdrive is offering 3 certs that expired 9/30/2021 along with the Lets Encrypt cert (not exactly a best practice.)
The older version of curl in Mojave (7.54) (I'm guessing) failed on the expired certs never getting to the good LE cert.
Is it possible to release an updated version for those of us who don't understand enough about how to implement the apparently simple fix outlined above? I gather I need to edit overdrive.sh but all I see is the executable file at /usr/local/Cellar/overdrive/2.1.1/bin/overdrive and I'm too clueless to know how to edit it. Sorry for being so helpless.
For what it's worth, I've got one Mac running Mojave and another running Big Sur. I can download fine on the Mac running Big Sur, which has curl 7.64.1, but not with the one running Mojave and curl 7.54.
Ok. Hi all! I'm a Newbie here who knows nothing about coding so I really need my hand held. Does this work around work with Big Sur 11.6.1? If so, what are the step by step directions. If it works with Big Sur are there any changes to the terminal coding? I switched my user agent to Google Chrome Windows, got my mp3 audiobook download button to appear, downloaded the audiobook, ran the installation step and then came the first error fix:
At this point, if calling overdrive produces the error message -bash: overdrive: command not found, you'll need to add ~/.local/bin to your PATH. One way to do this:
printf 'export PATH=$HOME/.local/bin:$PATH\n' >> ~/.bashrc
source ~/.bashrc
Wasn't sure what to do with the above instructions. I didn't want to mess around too much. I did try a few times with the same book loan.
I'd love for this to work for me. I want to still be able to listen to my audiobooks on my 6th generation iPod but to do this I need to continue to download the MP3 Overdrive file to my MacBook Pro and drag it into my Apple Music (aka iTunes).
Thanking is advance for this wonderful program you created and for any help. Happy Holidays!
Does this work around work with Big Sur 11.6.1?
It should, though the previous commenter noted that Big Sur didn't need the workaround at all.
(Personally, I haven't yet upgraded to Big Sur, so can't say anything from first-hand experience.)
Note that if you're on a fresh install (not an upgrade) of Big Sur, you've probably got zsh as your default shell, so pay attention to the zsh parts of the README.
I switched my user agent to Google Chrome Windows, got my mp3 audiobook download button to appear, downloaded the audiobook, [...]
Oh, good thinking! That's a handy step I really ought to document in the README 😄
[...] then came the first error fix:
At this point, if calling overdrive produces the error message -bash: overdrive: command not found, you'll need to add ~/.local/bin to your PATH.
That's just a quote of the documentation from the README. I don't know if it's relevant to your case. Honestly I don't even know if you actually ran into any problems — you don't explicitly say that, but I'm just kinda assuming something went wrong since you're posting here.
If you are running into that -bash: overdrive: command not found error message in your terminal: first, note that your shell is bash, not zsh. Second, when the README says "One way to do this:" that means that, to fix it, you copy & paste that entire code block into your terminal. (And you might need to execute the final command manually if you didn't happen to capture the final newline when selecting the code block.)
Otherwise, you're gonna have to come back with a lot more details, including:
- What have you done so far leading up to the error
- What does the error say
- What have you tried to fix the error
P.S. There's a YouTube tutorial at https://youtu.be/8Lgnh9To7Fw that might be helpful? I haven't gone through the whole thing — I didn't make it — but it may help you see what the process looks like end-to-end.
Hi. Here is a step by step of what I tried. Hope this is enough detail:
I think the default shell is my issue. If I can get this right I'm hoping it will work for me:
PUBLIC LIBRARY:
Borrow Book from LIBRARY LOANS
Switch Agents to access “download button” on Safari
Click Download Audiobook Button : File goes to download file folder as xxxaudiobook.odm
Bring up Github App: https://github.com/chbrown/overdrive#instructions and/or
https://www.youtube.com/watch?v=8Lgnh9To7Fw
TERMINAL:
The default interactive shell is now zsh.
To update your account to use zsh, please run `chsh -s /bin/zsh`.
For more details, please visit https://support.apple.com/kb/HT208050.
EllenMaookPro15:~ ellenstock$ Pasted the following code block above:
mkdir -p ~/.local/bin
curl https://chbrown.github.io/overdrive/overdrive.sh -o ~/.local/bin/overdrive
chmod +x ~/.local/bin/overdriveQUESTION: I understand I only need to do this one time for all audiobooks?
Got the following:
The default interactive shell is now zsh.
To update your account to use zsh, please run `chsh -s /bin/zsh`.
For more details, please visit https://support.apple.com/kb/HT208050.
EllenMaookPro15:~ ellenstock$ mkdir -p ~/.local/bin
EllenMaookPro15:~ ellenstock$ curl https://chbrown.github.io/overdrive/overdrive.sh -o ~/.local/bin/overdrive
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 8911 100 8911 0 0 80279 0 --:--:-- --:--:-- --:--:-- 80279
EllenMaookPro15:~ ellenstock$ chmod +x ~/.local/bin/overdriveCLOSED ABOVE TERMINAL WINDOW & OPENED NEW TERMIMAL WINDOW:
Last login: Thu Dec 30 21:15:19 on ttys000
The default interactive shell is now zsh.
To update your account to use zsh, please run `chsh -s /bin/zsh`.
For more details, please visit https://support.apple.com/kb/HT208050.
EllenMaookPro15:~ ellenstock$TYPE: overdrive download
DRAG: downloaded odm audiobook file from my desktop to the terminal
HIT: enter
RESULTS:
Last login: Thu Dec 30 21:15:19 on ttys000
The default interactive shell is now zsh.
EllenMaookPro15:~ ellenstock$ overdrive download/Users/ellenstock/Desktop/Origin
EllenMaookPro15:~ ellenstock$ overdrive download/Users/ellenstock/Desktop/Origin
EllenMaookPro15:~ ellenstock$ overdrive download/Users/ellenstock/Desktop/Origin_9780739319307_2968609.odm
-bash: overdrive: command not found
EllenMaookPro15:~ ellenstock$ GOT THE ERROR MESSAGE
-bash: overdrive: command not found QUESTION: Why does it say “The default interactive shell is now zsh.” When the terminal window says myname – chsh < -bash – 80x12”?
QUESTION:
So, does this mean my default shell is bash and I need to change the DEFAULT SHELL from zsh to bash? It’s confusing to me.
THE README SAYS:
At this point, if calling overdrive produces the error message -bash: overdrive: command not found, you'll need to add ~/.local/bin to your PATH. One way to do this:
printf 'export PATH=$HOME/.local/bin:$PATH\n' >> ~/.bashrc
source ~/.bashrcQUESTION: Do I copy and past the above code block into a new terminal window?
COPIED RESULTS COPIED ABOVE CODE BLOCK IN NEW TERMINAL WINDOW:
RESULTS:
Last login: Thu Dec 30 21:21:44 on ttys000
The default interactive shell is now zsh.
To update your account to use zsh, please run `chsh -s /bin/zsh`.
For more details, please visit https://support.apple.com/kb/HT208050.
EllenMaookPro15:~ ellenstock$ printf 'export PATH=$HOME/.local/bin:$PATH\n' >> ~/.bashrc
EllenMaookPro15:~ ellenstock$ source ~/.bashrc
EllenMaookPro15:~ ellenstock$ QUESTION: Does the above results show the default shell is now BASH? Does this step make the default shell stay as BASH or do I have to do this step every time I borrow a book?
QUESTION:
"And you might need to execute the final command manually if you didn't happen to capture the final newline when selecting the code block." How and where do I do this?
I feel all I need is a simple tweek to the default shell for this to work for me. I just still don’t get how to get there. Am I leaving out important steps? Hope this makes some sense.
I googled how to change the default shell and there are several ways. They all seem to need my password and then to reboot the computer. Is this another way? I did not want to do this since it’s nowhere in the README ON THE GITHUB instructions.
Thank you for your time and HAPPY NEW YEAR!
I think the default shell is my issue.
Wish it were that easy, but I doubt it. zsh is a fork of bash and everything we use here works exactly the same way, they just look for certain settings in different files, e.g. .bashrc vs .zshrc.
QUESTION: I understand I only need to do this one time for all audiobooks?
Yes.
QUESTION: Why does it say “The default interactive shell is now zsh.” When the terminal window says myname – chsh < -bash – 80x12”?
Dunno. But you're definitely in bash, which is fine. That's what I use, and that's what the instructions primarily target.
QUESTION:
So, does this mean my default shell is bash and I need to change the DEFAULT SHELL from zsh to bash? It’s confusing to me.
It's saying the default is different than your current shell. Ignore it.
QUESTION: Do I copy and past the above code block into a new terminal window?
Yep. Doesn't have to be a new window but that's fine.
QUESTION: Does the above results show the default shell is now BASH? Does this step make the default shell stay as BASH or do I have to do this step every time I borrow a book?
No, but don't worry about it. Just let it stay bash. The important thing is the error message, which clearly starts -bash: [...].
QUESTION: "And you might need to execute the final command manually if you didn't happen to capture the final newline when selecting the code block." How and where do I do this?
Just hit "enter" on your keyboard, but I can see from the "RESULTS" above that it ran.
I feel all I need is a simple tweek to the default shell for this to work for me. I just still don’t get how to get there. Am I leaving out important steps? Hope this makes some sense.
I think you are there :) — it looks like you have everything set up properly at this point, so just run the overdrive download ~/Desktop/Origin_9780739319307_2968609.odm command again.
P.S. if you run into any more errors, it makes it a lot easier to read your comments if you use code fences in your message — TIA. I edited your last comment to make it easier to follow, see those changes for reference.
Regarding the original issue (last 4 comments above are totally off-topic), I can't reproduce with Homebrewed curl on up-to-date Catalina now, so maybe they fixed it upstream?
If not, I added support for an --insecure option, which simply causes each invocation of curl [...] to call curl --insecure [...] instead. This isn't a fix, per se, but makes the workaround much easier — just call overdrive download MyBook.odm --insecure or whatever.
Again, thanks for your time and attention. I realize your help is as only good as what I am trying to show you and that's not really fair to you as you probably aren't getting the entire picture.
That being said,I tried the process again with a fresh loan of Giver of Stars and added the --insecure command. Got this:
EllenMaookPro15:~ ellenstock$ overdrive download/Users/ellenstock/Downloads/TheGiverofStars_9780525530206_4581755.odm --insecure
-bash: overdrive: command not found
EllenMaookPro15:~ ellenstock$
You're very generous with your time thanks again! Happy New Year!
[...] Got this:
EllenMaookPro15:~ ellenstock$ overdrive download/Users/ellenstock/Downloads/TheGiverofStars_9780525530206_4581755.odm --insecure -bash: overdrive: command not found EllenMaookPro15:~ ellenstock$
@Elsto007 the --insecure bit isn't your problem — I was addressing what this issue was originally about. You have a different problem, so you should have created a new issue, but whatever...
Your problem now is the same one that like 90% of 'Newbie' users run into, in light of which I just totally revamped the README. So please jump back to the main page (https://github.com/chbrown/overdrive), refresh, and follow the new and much-simplified Instructions section from top to bottom. Then, if you're still stuck, please create a new issue.
And again, code fences would be cool. Thanks!
Hi Christopher,
SUCCESS! The revamped README really helped. Thanks for sticking with me. As a "newbie" I really appreciate your being extremely patient with me.
I even learned about code fences. 😊
Happy New Year!
Ellen