ICMP Type 3 messages being blocked on an already open connection.

[BoatMisser]

What happened:
A call to an external SOAP service kept timing out; when using curl it returned a no response error message.
Once I worked out how to enable the flow logs (instructions missing from the online configuration guide) I saw this:

{"level":"info","ts":"2024-02-07T19:07:00.273Z","logger":"ebpf-client","msg":"Flow Info:  ","Src IP":"100.64.44.16","Src Port":48530,"Dest IP":"57.20.37.65","Dest Port":8443,"Proto":"TCP","Verdict":"ACCEPT"}
{"level":"info","ts":"2024-02-07T19:07:00.513Z","logger":"ebpf-client","msg":"Flow Info:  ","Src IP":"57.20.37.65","Src Port":0,"Dest IP":"100.64.44.16","Dest Port":0,"Proto":"ICMP","Verdict":"DENY"}

Once I saw the ICMP DENY message I was confused as this is not an Ingress connection; it is on the response to the already established Egress connection over which the SOAP request was sent.
When playing around with the NetworkPolicy definition for this app I found that the ICMP is accepted when I open an Ingress rule for the IP with no protocol or port restriction.

When using wireshark to take a network packet snapshot I saw the ICMP message is a Type 3 - Fragmentation needed.
I have been assured by our Networking team that such packets should be allowed over an open connection.

So, my issue is why is this ICMP Type 3 message being treated as a new Ingress when it is not; and why these messages are blocked when it seems they should be allowed?

Attach logs

I could not get that Log Collection tool to run on our Bottlerocket node.

What you expected to happen:
The ICMP Type 3 message should be allowed to reach the pod as it is coming in on an already established Egress connection.

How to reproduce it (as minimally and precisely as possible):
Make a call to a service that sends the following ICMP Type 3 message:

Internet Control Message Protocol
    Type: 3 (Destination unreachable)
    Code: 4 (Fragmentation needed)
    Checksum: 0xc2e1 [correct]
    [Checksum Status: Good]
    Unused: 0000
    MTU of next hop: 1372

Anything else we need to know?:
Previously we used Calico and with the exact same Network Policy definition and it works; using wireshark we can see the ICMP message reaching the pod. So Calico's implementation allows ICMP Type 3 messages on an already open Egress connection.

I also have an open AWS support case about this; they were the ones who suggested raising a bug report here.

Environment:

• Kubernetes version (use kubectl version): v1.28.5-eks-5e0fdde
• CNI Version: v1.16.2-eksbuild.1
• Network Policy Agent Version: v1.0.7-eksbuild.1
• OS (e.g: cat /etc/os-release): "Amazon Linux 2"
• Kernel (e.g. uname -a): 6.1.61

[Apurup Chevuru]

@boatmisser Right now, there is no way to explicitly allow ICMP traffic via K8S Network policy resource. List of protocols are limited to TCP, UDP and SCTP. For reference ..

For other protocols, such as ARP or ICMP, the behaviour is undefined. The same applies to allow rules: when a specific pod is allowed as ingress source or egress destination, it is undefined what happens with (for example) ICMP packets. Protocols such as ICMP may be allowed by some network plugins and denied by others.

We decided to limit it to the above list of allowed protocols in our implementation and to allow ICMP one needs to allow all protocols. But, I believe your scenario is a very valid use case and I think we can allow ICMP for already established connections. We will address this use case and will keep this thread updated once we've a potential solution.

[Jeffrey Nelson]

xref: #90