HTTP errors swallow potentially useful response content

[Christopher Brown]

The --fail flag in curl -o my.out --fail ... handily avoids writing invalid responses to my.out when the HTTP status code is an error, but unfortunately also causes the response body to be completely omitted, making it difficult to determine what the problem was.

AFAIK, there is no combination of curl flags to treat response content distinctly differently based on the returned HTTP status being an error or not. Ideally, I'd like to:

1. On HTTP 200 OK (or any 2** code, ideally), write response body to given output file and return exit code 0 so that the script (running with set -e) continues apace
2. On any error, print response body to stdout (or stderr, whichever), do not write to the given file, and return a non-zero exit code so that the script exits

That's pretty much what currently happens with -o ... --fail, except for the bold bit.

[gtTracy]

I haven't done much looking (as I am currently at work), but I did stumble across this thread which *seems* to cover this (again I gave it only a quick read, so apologies if I missed a turn)... https://superuser.com/questions/272265/getting-curl-to-output-http-status-code Specifically, I was looking at the response posted by Heath Borders

…

On Sat, Feb 5, 2022, 13:42 Christopher Brown ***@***.***> wrote: The --fail flag in curl -o my.out --fail ... handily avoids writing invalid responses to my.out when the HTTP status code is an error, but unfortunately also causes the response body to be completely omitted, making it difficult to determine what the problem was. AFAIK, there is no combination of curl flags to treat response content distinctly differently based on the returned HTTP status being an error or not. Ideally, I'd like to: 1. On HTTP 200 OK (or any 2** code, ideally), write response body to given output file and return exit code 0 so that the script (running with set -e) continues apace 2. On any error, *print response body to stdout* (or stderr, whichever), do not write to the given file, and return a non-zero exit code so that the script exits That's pretty much what currently happens with -o ... --fail, except for the bold bit. — Reply to this email directly, view it on GitHub <#46>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH73I4A7TGDHTGRE7N64PN3UZVVQNANCNFSM5NUJJ7BA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Christopher Brown]

Interesting, I wasn't familiar with curl's -w|--write-out flag. I'll have to play around with it a bit, but that might facilitate a solution like:

HTTP_CODE=$(curl -o my.out -w '%{http_code}')  # n.b.: no --fail
if [[ $HTTP_CODE != 2?? ]]; then
  >&2 cat my.out
  rm my.out
  exit 22  # what curl would have exited with if we'd told it to --fail
fi

A bit of a race condition there that doesn't quite achieve the same thing as --fail, but should solve #4 just as well, and even the curl manpage says, wrt. -f|--fail:

This method is not fail-safe and there are occasions where non-successful response codes will slip through, [...].

I'll dogfood it a bit locally first (as applied to the license issue in #4), but seems like this approach ought to work — thanks for the pointer, @gtTracy!

[Christopher Brown]

Looks like curl's planning to add this feature as a new --remove-on-error flag (curl/curl#8503), but probably not the best idea to depend on a cutting-edge addition when a lot of users are likely to just have whatever version of curl macOS builds in (which for up-to-date Catalina is 7.64.1, which was released on 2019-03-27).