Disabling Network Policy Enforcement After Enabling It

[Nick]

Hello!

I was trying to find any sort of documentation on how you would disable network policy enforcement via the CNI after you've enabled it.

I know to enable the network policy enforcement via the CNI, you can set the following configuration on the CNI:
{ "enableNetworkPolicy": "true" }

We are wondering if there's a way to "backout" having the CNI enforce the network policies in the cluster (in the event something were to happen after we began enforcing them).

I attempted in our testing environment to update the addon and set:
{ "enableNetworkPolicy": "false" }

And that begins to rotate out the aws-node pods in the cluster, but on the first one attempting to come up the aws-eks-nodeagent container goes into a crash backoff loop:

2024-01-24 16:50:29.677598122 +0000 UTC Logger.check error: failed to get caller
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x55cbd278b108]

goroutine 43 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:116 +0x1e5
panic({0x55cbd3452be0?, 0x55cbd48b73e0?})
        /root/sdk/go1.21.3/src/runtime/panic.go:914 +0x21f
github.com/aws/aws-network-policy-agent/controllers.(*PolicyEndpointsReconciler).configureeBPFProbes(0xc000212000, {0x55cbd37dfd38, 0xc0006e91a0}, {0xc00004c678, 0x17}, {0xc00062ef00?, 0x1, 0xc000454240?}, {0xc00073b380, 0x2, ...}, ...)
        /workspace/controllers/policyendpoints_controller.go:258 +0x3c8
github.com/aws/aws-network-policy-agent/controllers.(*PolicyEndpointsReconciler).reconcilePolicyEndpoint(0xc000212000, {0x55cbd37dfd38, 0xc0006e91a0}, 0xc0007bcd00)
        /workspace/controllers/policyendpoints_controller.go:232 +0x7f0
github.com/aws/aws-network-policy-agent/controllers.(*PolicyEndpointsReconciler).reconcile(0xc000212000, {0x55cbd37dfd38, 0xc0006e91a0}, {{{0xc000460880, 0x7}, {0xc000468b60, 0x1b}}})
        /workspace/controllers/policyendpoints_controller.go:149 +0x22f
github.com/aws/aws-network-policy-agent/controllers.(*PolicyEndpointsReconciler).Reconcile(0xc000212000, {0x55cbd37dfd38, 0xc0006e91a0}, {{{0xc000460880, 0x7}, {0xc000468b60, 0x1b}}})
        /workspace/controllers/policyendpoints_controller.go:130 +0x125
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x55cbd37e1f08?, {0x55cbd37dfd38?, 0xc0006e91a0?}, {{{0xc000460880?, 0xb?}, {0xc000468b60?, 0x0?}}})
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0004408c0, {0x55cbd37dfd70, 0xc00061a3c0}, {0x55cbd3547580?, 0xc00060ca40?})
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316 +0x3c5
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0004408c0, {0x55cbd37dfd70, 0xc00061a3c0})
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266 +0x1c9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 105
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:223 +0x565

Is there a more proper way to stop having the CNI enforce the network policies?

Thanks!

Environment:

• Kubernetes version (use kubectl version): v1.25.16-eks-8cb36c9
• CNI Version: v1.15.4-eksbuild.1
• Network Policy Agent Version: v1.0.6-eksbuild.1
• OS (e.g: cat /etc/os-release): Amazon Linux 2
• Kernel (e.g. uname -a): 5.10.199-190.747.amzn2.x86_64

[Nick]

I think this is a duplicate of #135.

[Nick]

Going to close this one out. For anyone who happens to stumble on this, @jayanthvn did a good write up on how to disable enforcing the network policies after enabling them (if needed) in the linked issue above:

1. Delete all NP resources
2. Set enable-network-policy-controller to false in ConfigMap amazon-vpc-cni (kube-system NS). This will disable the controller.
3. Set the 'enableNetworkPolicy' parameter to false. This will disable the agents on the nodes.

We had tested this in our environments as well with no issues.