Support SSLKEYLOGFILE in SslStream
bgrainger opened this issue · comments
SSLKEYLOGFILE is a feature provided by Chrome and Firefox that logs the pre-master secret to a file (specified by the SSLKEYLOGFILE environment variable) during TLS negotiation. The format of the file is documented here: https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Key_Log_Format.
This allows encrypted data in a packet capture to be decrypted by Wireshark (without having to know the server's private key): https://wiki.wireshark.org/TLS#Using_the_.28Pre.29-Master-Secret.
As HTTPS and HTTP/2 become more popular, it will be increasingly more useful to be able to decrypt packet captures for network analysis; as per http://gary-nebbett.blogspot.com/2018/06/tracing-https-traffic-on-microsoft.html. (This use case also seems to be implied by #35369.)
The ability to use a network trace analysis tool is especially useful when HTTP/2 is in use because the binary encoding of HTTP/2 can easily be decoded and nicely presented by such tools.
It would be great if TLS negotiation performed by SslStream could also log these secrets so that tools like Wireshark could be used to decrypt packet captures involving .NET clients. (Note that I'm not necessarily asking for the SSLKEYLOGFILE environment variable to be specifically supported, but just some opt-in way of dumping the same data; SSLKEYLOGFILE seemed like the most concise way to describe the feature.)
Besides HTTPS, another use case would be TLS negotiation in a different protocol; for example, the MySQL protocol upgrades from plain text to TLS, and being able to decrypt the packets in Wireshark would make diagnosing issues easier: mysql-net/MySqlConnector#780 (comment)
One potential problem is that Schannel doesn't support exporting this information, which could make it difficult to implement this feature on all platforms: https://social.technet.microsoft.com/Forums/en-US/4041d78a-21bb-44fd-9a96-6579ea8129d1/obtaining-sslkeylogfilelike-data-from-edge-et-al-schannel-clients?forum=messageanalyzer
Tagging subscribers to this area: @dotnet/ncl
Notify danmosemsft if you want to be subscribed.
Triage: Useful for diagnostics to be able to view the traffic unencrypted.
Just as an aside, I foolishly assumed that given libcurl is a dependency when bootstrapping .net core in Linux, that you could export SSLKEYLOGFILE as an environment variable (see docs). Sadly, I can confirm this does not generate a key log file. So this feature would be very helpful.
Update: apparently SSLKEYLOGFILE started working overnight. Using Azure App Service, custom Docker image based off stopped working again... after a restartmcr.microsoft.com/dotnet/core/aspnet:3.1.8-alpine3.12
While SSLKEYLOGFILE may be problematic because of forward secrecy and platform dependency, I was experimenting with alternative approach. https://github.com/wfurt/PcapStream gives you option to create decrypted captures with HttpClient, Kestrel and perhaps other scenarios. It is in early stage but any feedback would be appreciated.
The MSQL use case is still problematic AFAIK as the client frames encrypted bits to TDS. I don't know Wireshark can deal with that. I did not have luck.
Native support in the runtime would be great. In the meantime, I've created a solution for Linux with OpenSSL 1.1.1+ (i.e. Ubuntu 18.04 up on amd64 or 20.04 all archs) that works by wrapping SSL_new and dlsym. A basic LD_PRELOAD substitution doesn't work because the .NET runtime provides an open handle to dlsym, so dlsym needs to be wrapped as well.
I understand this is a sensitive topic, but as TLSs is a common interface between organizations this feature is used to resolve disputes. IE: "See we are sending X but your are responding with Y and we expect Z. You can reproduce this collection procedure at your end as well to verify."
The idea that both parties can collect a common pcap, decrypt with their own SSLKEYLOGFILE, and reach consensus is commonplace.
Is there any ongoing work on this?
not really. We agreed to support it in DEBUG build in way similar to #83001. But the urgency is lower IMHO as there are other alternatives.
@wfurt I am very concerned about letting this option available only for debug builds. There is a very high risk of seeing debug code being deployed in production which is problematic for other reasons.
We already have secrets in the environment variables and this new variable has the same risk level.
Also, the provided workarounds are briliant but do not provide a solution for all the cases and require rebuilding which is something that can be hard to ask in large organizations.
When I said DEBUG I mean debug build for runtime. That is not published to usual feeds so there should be no confusion IMHO. Of course, somebody can work hard and mess it up anyway but it seems like this is what browsers do so we would be on par. (for Linux)
The rebuild is somewhat intentional. To limit risk you mentioned we down't want to make it too simple. You can always build it always and have App specific overrides to enable it - if that meets policy of give organization.
@wfurt just to be sure:
- do you mean a DEBUG build of the runtime that is on a different download repository?
- is this enabled on the sdk by default (which for debug purposes by design)?
- Are those debug builds available through the VS installer as well? (I ask this because the updates continuously uninstall/reinstall new releases from the main feed. This is done also from WU for security patches)
- Is this limited to QUIC (given the title of the PR) or for anything TLS related?
To limit risk you mentioned we down't want to make it too simple
I understand the concerns, but if I think to the container workloads, should the environment variables be defeated, security is already gone.
The PR is just for Quic, SslStream change can mimic that for TLS in future. And there are no debug binary bits available for download AFAIK. (but you can grab them for example from PR runs)
So the goal is to make it possible for somebody who is determined and knows what to do. Ease of access and wide availability is not priority at the moment.
Got it, thank you
Is there any update on this?
This feature is also very meaningful for gRPC on .NET. In most enterprise scenario, the gRPC traffic should be encrypted by TLS. A method to decrypt the TLS traffic is necessary for analyzing the streaming data. Whilst according to WireShark Wiki , currently C#/dotnet does not support TLS session secret key from either client or server side, while other languages like Java/Golang do support it.
It's better for .NET Core to support an easier way of decrypting HTTPS traffic, like exporting the TLS session secret key or any other method.
@hannadeng I don't think that this would be helpful in your case (gRPC).
Unfortunately this debug feature is QUIC only and only works if you get the runtime from a different feed.
For me it is a no-go.
Thanks @raffaeler . Maybe my question is not quite clear. Let me explain more. I understand #83001. is only for QUIC and will not be applicable in my case (gRPC - HTTP2 protocol). Whilst I thought the original issue requested by bgrainger is a kind of general request for supporting decrypting TLS, including HTTP/2 protocol. Appreciate @JamesNK could help input more insights.
@hannadeng I am totally with you. I would love to have the opportunity to monitor the SSL convesation with the required level of precautions needed to avoid misues.
Anyway in this thread I was said that they won't implement the decryption outside QUIC protocol.
You, me and many others are stuck, that's it.
@hannadeng I am totally with you. I would love to have the opportunity to monitor the SSL convesation with the required level of precautions needed to avoid misues. Anyway in this thread I was said that they won't implement the decryption outside QUIC protocol. You, me and many others are stuck, that's it.
Got it. Thanks again!
When #83001 is done I plan to do it for SslStream as well - Debug build on Linux only. But I don't have much time for either one at the moment.
If there a reason why https://github.com/wfurt/PcapStream does not work for you @hannadeng ? It should give you ability to decode and debug HTTP/2 and gRPC.
When #83001 is done I plan to do it for SslStream as well - Debug build on Linux only. But I don't have much time for either one at the moment.
If there a reason why https://github.com/wfurt/PcapStream does not work for you @hannadeng ? It should give you ability to decode and debug HTTP/2 and gRPC.
Thank you @wfurt , let me try this.
Bump.
Please get this feature asap. It will be very helpful in debugging.
There is too much on our plate at the moment for 8.0 and only a little bit time left (1 month for fixing all known bugs + features should be finished by Monday), so the feature will most likely not make it in 8.0 - hence the milestone Future.
In the meantime, can you use the workaround / PoC which @wfurt mentioned and asked about? https://github.com/wfurt/PcapStream
Tentatively re-opening to discuss the requirement for DEBUG build (as this no longer applies to browsers - Firefox, Chromiun, nor CLI tools - cUrl)
And to make sure we implement this for Windows as well, see #94843
I would like to know if there is any chance to discuss supporting SSLKEYLOGFILE in release builds of the CLR, using an environment variable to enable easier diagnostics.
To support this request:
- Even on dev machines, I doubt that the majority of devs would install a debug build. In my experience they would rather switch to plain http to make the test and hopefully restore back https
- Env variables are already used to hide secrets, this is no different
- When using
dotnet/aspire, we can request a future feature triggering a warning whenever the env variable is turned on potentially exposing the distributed app to SSLKEYLOGFILE
If this is out of discussion for .NET 9, please make a statement so that we avoid 'noise' in this comments.
Thanks!
I would like to know if there is any chance to discuss supporting SSLKEYLOGFILE in release builds of the CLR, using an environment variable to enable easier diagnostics.
That's the intent here.
If this is out of discussion for .NET 9, please make a statement so that we avoid 'noise' in this comments.
No, I'd like to solve this in .NET 9.
And thank you for describing your scenario, that will help make my case.
Any updates?
Thanks for the remainder, I just kicked off an internal discussion about this.
Can you elaborate on why the SslStream cannot emit the Traffic or Handshake secret?
| OS | SslStream | QuicConnection |
|---|---|---|
| Windows | Not possible | Yes |
There is no API on Windows to extract the secrets @upsampled . SslStream does not implement TLS - it just wraps underlying OS functions and therefore it is dependent on the provided capabilities.
If Windows client is your only one option thee are other ways how to peek inside - something like Fiddler or PcapStream should still work.
Can you elaborate on why the SslStream cannot emit the Traffic or Handshake secret?
The underlying OS library (Schannel) does not give out secrets to the application. In fact, the secrets are not kept in the app process memory at all, but instead reside in lsass.exe process and there is IPC between the two under the hood. https://b.poc.fun/decrypting-schannel-tls-part-1/ may be an interesting read (although the method described may not be practical).
Schannel added support for exporting TLS 1.3 secrets specifically in order to support implementing QUIC (QUIC requires access to encryption secrets because it interleaves TLS + other protocol data + application data and applies custom encryption using the secrets derived by TLS layer). That support is intentionally limited to very specific use case where Schannel gives out unencrypted TLS messages and encryption secrets separately. IIRC, this mode does not work for TLS 1.2.
We could theoretically use that mode and apply TLS framing+encryption in .NET, but that has few problems:
- Only TLS 1.3 would be supported, version detection may be problematic because the leave the version detection/negotiation on Schannel.
- For security reasons, that code could be only used when SSLKEYLOGFILE functionality was desired in order to avoid accidental vulnerabilities in production code
- Connected to the above: there would be potentially observable difference between SSLKEYLOGFILE and normal behavior, thus making the feature potentially useless
All-in-all, the implementation cost far outweighs possible benefits and it might be better to use other tools like wfurt mentioned.
Schannel added support for exporting TLS 1.3 secrets specifically in order to support implementing QUIC
Are those same exporting mechanisms available for TLSv13 over TCP? I am assuming this is to support 0-RTT or the PSK-binder mechanisms in TLSv13 and both also are possible over TCP.
I guess I am asking if the below is true
| OS | SslStream (<=TLSv12) | SslStream (TLSv13) | QuicConnection (TLSv13) |
|---|---|---|---|
| Windows | Not possible | Possible | Yes |
Are those same exporting mechanisms available for TLSv13 over TCP?
No, let me clarify. The secrets are exported only if the TLS version is 1.3 and it is operating in "recordless mode" (where it strips outer encryption and provides unencrypted TLS messages).
SChannel does not know about TCP, it is simply exchanging data via buffers and sending data is up to other code.
Putting the stripped encryption back in .NET code is the complex and problematic part which is the main reason we won't invest into that, for reasons listed in my previous comment.
Can you elaborate on why the SslStream cannot emit the Traffic or Handshake secret?
OS SslStream QuicConnection
Windows Not possible Yes
In windows maybe you can consider using some OpenSSL wrapper instead of SslStream. Like OpenSSL Net does
Can you elaborate on why the SslStream cannot emit the Traffic or Handshake secret?
OS SslStream QuicConnection
Windows Not possible YesIn windows maybe you can consider using some OpenSSL wrapper instead of SslStream. Like OpenSSL Net does
we could. But that has massive implications for certifications, performance, security and integration with OS functions like Cert store. As I mentioned above, there are other ways how to solve debugging TLS traffic @nathan130200
It makes sense, and it ends up becoming unfeasible to do this at the moment, especially when we talk about performance.
If Windows client is your only one option there are other ways how to peek inside - something like Fiddler or PcapStream should still work.
PcapStream looks nice, but it is only usable if you are the app developer (not always the case, you might want to debug a third-party app/component).
Fiddler is only usable if the app supports proxies and does not implement SSL pinning or similar mechanisms. It does not help with non-HTTP connections as far as I know.
Imagine you are an app developer using a third-party component that implements some protocol (e.g. BACnet SC, SNMPv3, SRTP...) and you'd want to debug an issue in the communication in between your app using that component and an actual device. For this example let's consider a scenario when the third-party component does not expose the underlying streams to the developer directly, meaning we have to go through some API and we cannot use PcapStream.
If the component is using OpenSSL (with keylog for debugging), then it's quite easy: you configure the environment which you fully control to dump the keys, point Wireshark to that file and you'll see the decrypted traffic -> happy debugging.
However, if that third-party library is not using OpenSSL, then you're out of luck. Is the issue you are trying to debug in your code? Is it in the third-party component? Is it on the device? What's happening on the wire?
I think there should be some easy opt-in mechanism like the one in OpenSSL to allow developers to peek inside the TLS encrypted stream and see the decrypted traffic, provided they fully control at least one side of the connection. But I understand from the previous comments that maybe it's not an issue of .NET, but Windows API in general. Is my understanding correct?