metadata validation disallows UTF-8/ISO-8859-1 characters in headers
CosmicToast opened this issue · comments
Please see the FAQ in our main README.md, then answer the questions below
before submitting your issue.
A few questions omitted because the answers are complex (multiple machines and go versions involved).
Additionally, the source of the issue is well-known: #4886.
What did you do?
I sent a grpc call to argo-cd through a tailscale ingress.
Tailscale ingress inserted my name (which contains a UTF-8 character "é") in the tailscale-user-name header.
While this is discouraged in the HTTP RFC, to my knowledge, it is not invalid.
Of particular note, ISO-8859-1 is explicitly mentioned as allowable (if, again, discouraged).
What did you expect to see?
I expected the call to succeed, with at most a validation warning that the spec discourages the usage of non-ASCII (but still printable) characters, especially as this header is not used by grpc and could be safely ignored.
What did you see instead?
internal/metadata.hasNotPrintable returned true, leading to the entire request failing and me being unable to use the remote service.
A very common triggering scenario are ingress nodes adding additional metadata for downstream services.
For example, X-GeoIP-City and tailscale-user-name.
An example downstream bug caused by this is argoproj/argo-workflows#12721.
@CosmicToast It would help us if you let us know where is the call failing, is this on the send or the receive side?
Also per our spec, gRPC restrictions on the header values are more strict than of the HTTP spec. gRPC-go right now throws an error, while we should be ignoring and throwing away the header key-value.
However, there isnt a possibility of transmitting the value through the gRPC service.
It would help us if you let us know where is the call failing, is this on the send or the receive side?
It's the receive side.
The pipeline looks something like this:
- The sender makes a gRPC request over HTTP (either 1.1 and 2).
- This call is intercepted by a reverse proxy. This is for authentication and common reverse proxy use-cases. The vast majority of internet traffic goes through one, but enterprise environments tend to add headers...
- The reverse-proxy adds some headers for downstream consumers to optionally use. In my case, it's tailscale adding
tailscale-user-name. In the linked bug case, it's nginx adding X-GeoIP-City. - The message continues on to the receiver (in my case, argo workflows server), at which point it is rejected since a header contains a character from Latin Extended 1.
Also per our spec, gRPC restrictions on the header values are more strict than of the HTTP spec. gRPC-go right now throws an error, while we should be ignoring and throwing away the header key-value.
I agree!
That's my desired resolution.
Specifically, https://github.com/Patrick0308/grpc-go/blob/master/internal/metadata/metadata.go#L104 should not return an error but drop the header in question.
However, there isnt a possibility of transmitting the value through the gRPC service.
I'm not entirely sure what you mean in this respect.
Could you clarify your comment a little?
It seems to imply that the abovementioned scenario cannot possibly happen, but it quite clearly does.
As an update, tailscale should no longer be affected (a workaround for grpc users has been pushed).
This will continue to affect nginx users with x-geoip-city and similar headers, so I'm leaving this open for now.
If there's a definitive fix on that end as well then this can probably just be closed.
The receive side does not go through metadata.Validate codepath. This happens only on the send side. Which means we do not error if the server receives a header with characters outside the ASCII range.
I double checked this ^^ using this test https://github.com/arvindbr8/grpc-go/blob/7cb635a70fcfc8dc6cf71c71554ff8cac4326568/test/end2end_test.go#L922
a workaround for grpc users has been pushed
Curious to know what the workaround here is
Hmm, in that case that sounds like an argo bug indeed.
That means that the bug is fundamentally caused by the argo cd devs copying the http request they receive verbatim and retransmetting it via grpc.
I'm terribly sorry about the misunderstanding, their sources were a little hard to read, but I did manage to follow the trail all the way to the abovementioned line.
Given this, I'll bring it back up with them.
Curious to know what the workaround here is.
Tailscale will mime-qencode non-us-ascii header values.
This means that even services like argo that blindly copy over the http request will now work with it.
You can find the details in the mentioned issue: tailscale/tailscale#11603