Retry Offload might drop second initial
martinduke opened this issue · comments
There are cases where an offload might send Retry on the second initial but not the first:
- it turns on between the two packets
- it is in a stochastic mode where it doesn't always send Retry
There is one failure mode and one degraded performance issue:
- If the server sends HRR, if the client's response triggers Retry it is likely to blackhole the connection.
- Under certain conditions, dropping the client's second Initial packet will cause either a client or server PTO. (Specifically, there will be no penalty if the client's 2nd flight has a datagram that (a) does not contain an initial, and (b) contains a HS ACK.)
We ought to write this down in the draft. For #1, it might be good practice for servers to send CONNECTION_CLOSE instead of HRR under two conditions:
- There is no Retry token in the Initial, and
- Other recent initials contained Retry tokens.
The amplification limit is a problem here. If the client's second flight is one packet, including an Initial, which keeps getting dropped, and the server is at its amplification limit, there is no way for the connection to progress.
As I mentioned in OR #170, I think we need to handle this via an extension to the QUIC protocol itself. I recommend we define an extension that does the following:
The server includes a new transport parameter initial_token that contains the value of the initial token that clients should use for any subsequent initial packets they send. If the client understands this extension/TP, it acts accordingly. If it does not, then it may hit the issues you describe in this PR and eventually time out.
So the problem with using a TP is that it isn't delivered until the Encrypted Extensions messages, which is far too late for most of the failure modes here.
However, I've thought more carefully about this and have realized some things.
-
Offload statefulness doesn't completely solve it, as you could have a shared-state retry where an anti-DDoS service turns on and suddenly has all packets route through it.
-
If an offload is going to randomly send Retry while being stateless, it really ought to be pseudorandom based on the address/port or whatever. Therefore, the two Initial packets will get the same treatment in the usual case. That reduces to the problem to "service turns on mid-handshake".
-
Case 1 is where Retry offload breaks the Initial + Handshake flight. This isn't actually a black hole. The client will retransmit handshake packets and, per RFC9001 4.9.1, MUST NOT send Initial packets once it's sent a handshake. So there might be a PTO, which is not ideal, but not catastrophic. In the normal case, there are multiple server handshake packets, and if the client is following spec and not doing ack delay in the handshake, it is unlikely the entire Handshake ACK will be coalesced with Initial. So let's forget about this "problem", and we don't need to do any weird packet truncation as in PR #170.
-
Case 2 is HRR or Multi-packet SHLO: this is the harder case as Initial ACKs are critical to progress. HRR is an edge case but post-quantum we may need multi-packet SHLO, and it would be good for this design to be robust to that. I don't see a realistic fix here right now but I'll think about it some more.
So the problem with using a TP is that it isn't delivered until the Encrypted Extensions messages, which is far too late for most of the failure modes here.
Crap. You're obviously totally right. I don't know how I forgot. I still thing the right way to handle this is to somehow give the client the initial_token along with the server's initial payload. I might propose a slight modification of my original plan:
The client includes a new transport parameter initial_token (with no payload) that advertises support for this new extension. If the server supports this extension, it includes a initial_token frame in all its Initial packets, and then also includes the initial_token transport parameter that also contains the value of the initial token (so that it can be securely validated). Clients should use the initial_token frame value for any subsequent initial packets they send.
That will work, but I am very skeptical of client support, particularly in Chrome.
I just reworked things completely with #172. Reasoning about the Handshake is hard, but I think this works pretty well unless the handshake is also suffering losses for reasons other than the Offload dropping stuff.
Basically, there's a "transition mode" when going active where you send Retry and forward the packet to the server anyway. The handshake logic basically works out, which is surprising until you think it through.
I'm very hesitant to push a solution that doesn't work in all scenarios. From some tests I've done in recent past, HRR is a lot more common in some scenarios. Also, post-quantum crypto is coming.
But this works!