SecureTransport segfault when doing an HTTPS GET in a forked child process
julik opened this issue · comments
I did this
Performed a GET request on an HTTPS URL in a forked child process on OSX. When doing so, libCURL crashes in preference-related xpc call. This is specific to Darwin/SecureTransport and only happens if the request is performed in the forked child process, not in the master.
I expected the following
I expected the download to complete without errors, but it segfaults instead. Just like https://sourceforge.net/p/curl/bugs/1446/ describes. Crashdump: https://gist.github.com/julik/79129b30275b275b30fcf79e8e99f854
Related issues in other projects (all libCURL dependents, all show the issue):
taf2/curb#239 (comment)
taf2/curb#202
aws/aws-sdk-php#737
facebook/hhvm#6189
toland/patron#72
I think this previous issue did not mention the crucial repro detail: the crash will only happen if you perform the request in a forked child process. I am positive in my case the CURL handle gets allocated within the child process and does not get copied. Consequently, all scripting environments/VMs that use forking (like web servers, web server application server adapters, job queue engines and so forth) are very likely to exhibit this issue.
curl/libcurl version
curl 7.43.0 (x86_64-apple-darwin15.0) libcurl/7.43.0 SecureTransport zlib/1.2.5
When building newer curl via homebrew the problem persists unless curl is forcibly switched to build against OpenSSL instead of SecureTransport. I.e. this one works:
curl 7.48.0 (x86_64-apple-darwin15.4.0) libcurl/7.48.0 OpenSSL/1.0.2g zlib/1.2.5
operating system
OSX 10.11 (Darwin 15.4.0) and 10.10, both observed.
That crash is coming from inside the Security framework; unfortunately, we can't do anything about it. Have you reported the problem to Apple? https://bugreport.apple.com/
Yes, please report this to Apple as we'd need someone with insight and knowledge about the secure transport internals and how it is supposed to deal with this. If you have source code for a way to reproduce the crash like against a public URL, I'm convinced that will help a long way with this!
My C is not good enough to set up a bare example and I've never reported radars before. Given that this causes actual segfaults, and nasty ones at that I think it might be a good idea for libCURL to protect the user in this case. For instance, fail perform() with a curl error saying that there is a bug in play. It's not pretty, but it can easily take half a decade before Apple fixes something of this nature, and all the default curl installs on OSX (including homebrew) are affected. Or see if this can be helped by changing options for CreateSSLContext. Or document this as a known issue.
So even though the temptation to "blame it all on Apple" is big (and well grounded) I think CURL should do something about this as well - simply because it subscribes Mac users to SecureTransport by default.
For instance, fail perform() with a curl error saying that there is a bug in play
When/how exactly would we do that?
it subscribes Mac users to SecureTransport by default
It actually doesn't. First, the project only ships source code so people actually pick the TLS backend they want. curl on mac still selects OpenSSL by default when built with configure. You need to explicitly ask for it to get used (./configure --with-darwinssl).
Apple, the Homebrew project and probably a few others, however, build and ship curl built with Secure Transport by default.
Or document this as a known issue.
Certainly, but I want more evidence and pointers that say this is actually a problem and the correctly stated problem before I try that...
Apple, the Homebrew project and probably a few others, however, build and ship curl built with Secure Transport by default.
That is the very issue which makes this a painful bug. Ok, I will try to make a minimal C example just using fork(), although this is a heavy "burden of proof" for a C-challenged person like me ;-) For now I think I will approach this from two directions: I will try to persuade homebrew folks to switch back to openssl, and also try to file a radar/openradar against the preferences reading CoreFoundation stuff that actually crashes SSLCreateContext, but for that I am going to need someone familiar with CoreFoundation to file it for me. And some familiarity with radar. And familiarity with SSLCreateContext.
Meanwhile, curl is crashing for people and the issue seems to be pretty common. It is very sucky to be in a position where a chain of dependencies leads to Cupertino, because I can't just go and submit a pull request against _CFPrefsWithDaemonConnection (or even set this up as a bounty), let alone know when it gets shipped via an OS update.
The details you've provided so far are actually possibly indicating that this is a problem in your application and not in Secure Transport, which is why I stress the reproducible code part. To me it seems you assume you can do things after a fork() that the environment doesn't allow you.
After the fork, the Darwin libc opts to crash instead of risking waiting for a lock it might not get since the child process did not inherit the pthread mutexes (and more) from the parent process. (told to me in private by someone who wanted to help out)
Ok, please don't judge my C too harshly, but I think this one shows the problem:
https://gist.github.com/julik/58c0b9713ac2d2372d799ca562281c9a
If you run it you will see that in child it will never get to the Request performed part but crash instead.
No threads.
Your code calls fork() prior to using libcurl. On Apple operating systems, if your code calls fork(), then it must call one of the exec family of functions, or you will get undefined behavior. This is documented in the fork() man page on OS X.
Remove the fork(), or call an exec function after the fork(), and it ought to work as expected.
This is not a problem with libcurl, so I'm closing this ticket.
if your code calls fork(), then it must call one of the exec family of functions
Since we are in the formal arguments territory from what I can see, this is also formally incorrect. This is a copy-on-write fork, an execution pattern also used by VMs. Whether it works by happy accident, or by conscious effort, it ends up being used by lots of projects. From this article http://stackoverflow.com/questions/18854708/why-is-it-prohivited-to-use-fork-without-exec-in-mac I can conclude that the same is true on Linux - http://www.linuxprogrammingblog.com/threads-and-fork-think-twice-before-using-them. Curl documentation does not specify whether it is fork-safe or not. Yes, formally you can say "this is undefined behavior, therefore we do not support it, therefore do not hit yourself with a hammer again when it hurts". Well done. Hear the keyboards click as people add compilation flags to all of their bindings.
This is not a problem with libcurl
That is the thing I am desperately trying to bring across here, but it just doesn't get through. Formally of course it isn't. Moreover, formally it is a crash in a third-party library that you have no control over.
However, real life is not as formally-defined as we would like. It is a problem if your users used not to experience issues with making your library do useful things before, and are experiencing one now because of a dependency swap that they did not opt in to themselves. This is amplified by the fact that by default even homebrew does not put a custom-built curl on $PATH since so many core system service shell out to it and link against it.
To summarize: formally it is not a problem with libcurl. Pragmatically (and holistically) it is a regression due to a dependency change, which creates actual issues for actual users.
You know, insulting people in GitHub issues is not something I do often - but the treatment of this problem has left me very disappointed. Basically you are blaming me of PEBKAC, whereas in reality your dependency does not support a usage pattern that most consumers of your library can get opted into by coercion. If someone's application server, or VM, uses CoW forking for process management, he should basically stop using curl altogether? If yes, then IMO it is worth at least a paragraph in the documentation.
@julik, What exactly are you suggesting we should do?
As I see it, this is a limitation in how OS X works. The crashing example works fine on Linux for example.
your dependency does not support a usage pattern that most consumers of your library can get opted into
This is BTW not a choice by us, but brought upon us by the ones who built-in those restrictions. And I wasn't aware they existed until you filed this issue. I would expect that most other libraries that try to do something similar to libcurl will end up having the same restrictions. I don't think we're unique.
@bagder ❤️ What if:
- Write up in the comments for the ST adapter that it is not CoW-fork-safe
- Put up a PR to the homebrew cask formula to switch back to OpenSSL by default. If I do it, it is not likely to be accepted. If it will be backed by you guys as an interim solution it stands a higher chance, so there is one build flag that can be omitted
- Set up another row of tests for SSL backends which check CoW-fork safety. Even a simple test like the one I posted would suffice to figure out if it crashes or not.
- In a broader sense, look at specifying behaviour in a CoW-fork context (it is possible that curl could be made signal-safe - because it does work fine with OpenSSL). Maybe a couple basic smoke tests in forked children could help.
That's a lot of things you'd want us to do.
The good thing here is that we're an open source project. We're all in this together. We're just one team. So consider yourself one player on the field, involved here just like any other player in team curl.
- I personally can't do that since as I said, I don't know enough of the specifics to help. I'll gladly accept pull-requests that clarify and help bringing light on this for other users. I would guess the "functionality after fork" details should better be a separate section in a man page somewhere.
- If step 1 is properly documented and explained, then any filed issue on homebrew could just refer to that and it shouldn't matter who submits said bug report. But I also suspect that a majority of OS X user use the bundled libcurl that comes with OS X and I presume that is not likely to switch away from Secure Transport.
- We're severely undermanned already. See our issues, pull-requests and the TODO and KNOWN_BUGS documents. We have things to do up to our ears for years to come. Adding more things to the TODO list is always fun and we can do that, but the best way to get things done is to step in and actually help making those things happen.
- isn't this limitation due to how Securetransport works? The limitation doesn't come from our glue code into the crypto libs, it comes from the crypto libs themselves. That's why it works with openssl.
- Put up a PR to the homebrew cask formula to switch back to OpenSSL by default. If I do it, it is not likely to be accepted. If it will be backed by you guys as an interim solution it stands a higher chance, so there is one build flag that can be omitted
Hey folks,
Homebrew maintainer here. Have been watching this thread since it was opened in case this was a limitation we could address on our side without user consequence. IRT Homebrew switching back to OpenSSL by default it's unlikely, particularly given the conclusion this is more or less expected behaviour on OS X when using Secure Transport.
We turn darwinssl on by default because:
- It doesn't inflict an OpenSSL dependency on people who just want an up-to-date
curlaround. - It behaves consistently with the system
curl, which is what most people want in our experience, and also reduces the maintenance burden on our side.
Secure Transport also handles certs a bit more sanely than OpenSSL, which I recently attempted to minimise via this commit but still remains a potentially problematic area outside of what Homebrew can control.
The mikeash article is a good read.
I think it would be a mistake to identify OpenSSL or anything else as a fork safe dependency.
You know, insulting people in GitHub issues is not something I do often
So don't? But I don't think anyone is going to take being disappointed as an insult.
@DomT4 will you be willing to accept a PR that warns about this behavior post-install when SecureTransport is linked? I think it could at least spare some grey hairs.
Hi, guys, good works to find it out! While keep the good work going, for any developers wanting to continue using spring with curb, here's the workaround:
brew reinstall curl --with-openssl
bundle config build.curb --with-curl-lib=/usr/local/opt/curl/lib --with-curl-include=/usr/local/opt/curl/include
gem uninstall curb
bundle install
# or if you don't use bundler
gem install curb -- --with-curl-lib=/usr/local/opt/curl/lib --with-curl-include=/usr/local/opt/curl/include
Edit: sorry I just realised that it's the curl repo. I should have put this in the ruby binding repo (curb) 's issue thread. I somehow followed an issued link here. Apologies. But still, thanks for the good work, and please, keep it going!