Disabling Network Policy Enforcement After Enabling It
ndrafahl opened this issue · comments
Hello!
I was trying to find any sort of documentation on how you would disable network policy enforcement via the CNI after you've enabled it.
I know to enable the network policy enforcement via the CNI, you can set the following configuration on the CNI:
{ "enableNetworkPolicy": "true" }
We are wondering if there's a way to "backout" having the CNI enforce the network policies in the cluster (in the event something were to happen after we began enforcing them).
I attempted in our testing environment to update the addon and set:
{ "enableNetworkPolicy": "false" }
And that begins to rotate out the aws-node pods in the cluster, but on the first one attempting to come up the aws-eks-nodeagent
container goes into a crash backoff loop:
2024-01-24 16:50:29.677598122 +0000 UTC Logger.check error: failed to get caller
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x55cbd278b108]
goroutine 43 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:116 +0x1e5
panic({0x55cbd3452be0?, 0x55cbd48b73e0?})
/root/sdk/go1.21.3/src/runtime/panic.go:914 +0x21f
github.com/aws/aws-network-policy-agent/controllers.(*PolicyEndpointsReconciler).configureeBPFProbes(0xc000212000, {0x55cbd37dfd38, 0xc0006e91a0}, {0xc00004c678, 0x17}, {0xc00062ef00?, 0x1, 0xc000454240?}, {0xc00073b380, 0x2, ...}, ...)
/workspace/controllers/policyendpoints_controller.go:258 +0x3c8
github.com/aws/aws-network-policy-agent/controllers.(*PolicyEndpointsReconciler).reconcilePolicyEndpoint(0xc000212000, {0x55cbd37dfd38, 0xc0006e91a0}, 0xc0007bcd00)
/workspace/controllers/policyendpoints_controller.go:232 +0x7f0
github.com/aws/aws-network-policy-agent/controllers.(*PolicyEndpointsReconciler).reconcile(0xc000212000, {0x55cbd37dfd38, 0xc0006e91a0}, {{{0xc000460880, 0x7}, {0xc000468b60, 0x1b}}})
/workspace/controllers/policyendpoints_controller.go:149 +0x22f
github.com/aws/aws-network-policy-agent/controllers.(*PolicyEndpointsReconciler).Reconcile(0xc000212000, {0x55cbd37dfd38, 0xc0006e91a0}, {{{0xc000460880, 0x7}, {0xc000468b60, 0x1b}}})
/workspace/controllers/policyendpoints_controller.go:130 +0x125
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x55cbd37e1f08?, {0x55cbd37dfd38?, 0xc0006e91a0?}, {{{0xc000460880?, 0xb?}, {0xc000468b60?, 0x0?}}})
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0004408c0, {0x55cbd37dfd70, 0xc00061a3c0}, {0x55cbd3547580?, 0xc00060ca40?})
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316 +0x3c5
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0004408c0, {0x55cbd37dfd70, 0xc00061a3c0})
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266 +0x1c9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 105
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:223 +0x565
Is there a more proper way to stop having the CNI enforce the network policies?
Thanks!
Environment:
- Kubernetes version (use
kubectl version
): v1.25.16-eks-8cb36c9 - CNI Version: v1.15.4-eksbuild.1
- Network Policy Agent Version: v1.0.6-eksbuild.1
- OS (e.g:
cat /etc/os-release
): Amazon Linux 2 - Kernel (e.g.
uname -a
): 5.10.199-190.747.amzn2.x86_64
Going to close this one out. For anyone who happens to stumble on this, @jayanthvn did a good write up on how to disable enforcing the network policies after enabling them (if needed) in the linked issue above:
- Delete all NP resources
- Set enable-network-policy-controller to false in ConfigMap amazon-vpc-cni (kube-system NS). This will disable the controller.
- Set the 'enableNetworkPolicy' parameter to false. This will disable the agents on the nodes.
We had tested this in our environments as well with no issues.