Enabling firewalld breaks podman network removal if pod creation fails
ccamacho opened this issue · comments
Enabling firewalld breaks podman network removal if ansible fails
When using containerized services, if a pod creation fails the pod network will be removed, in this case
the CNI firewall backend is not configured to firewalld, breaking Ansible with an unhandled exception.
TASK [../../roles/kubeinit_bind : Create BIND db internal zone] ****************
changed: [localhost -> 10.0.0.100] => {"changed": true, "checksum": "c626fda1426a6fa22269a87c5a8386761753938c", "dest": "/var/kubeinit/bind/zones/internals/db.rkecluster.kubeinit.local", "gid": 0, "group": "root", "md5sum": "9191bf51d97b4622def9fa987d3e1d1a", "mode": "0644", "owner": "root", "size": 1692, "src": "/root/.ansible/tmp/ansible-tmp-1627939723.4379623-3037-41765181103385/source", "state": "file", "uid": 0}
TASK [../../roles/kubeinit_bind : Create a podman container to serve the Bind server] ***
fatal: [localhost -> 10.0.0.100]: FAILED! => {"changed": false, "msg": "Can't pull image docker.io/internetsystemsconsortium/bind9:9.11", "stderr": "Trying to pull docker.io/internetsystemsconsortium/bind9:9.11...\ntime=\"2021-08-02T21:29:16Z\" level=warning msg=\"failed, retrying in 1s ... (1/3). Error: Error initializing source docker://internetsystemsconsortium/bind9:9.11: Error reading manifest 9.11 in docker.io/internetsystemsconsortium/bind9: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit\"\ntime=\"2021-08-02T21:29:49Z\" level=warning msg=\"failed, retrying in 1s ... (2/3). Error: Error initializing source docker://internetsystemsconsortium/bind9:9.11: Error reading manifest 9.11 in docker.io/internetsystemsconsortium/bind9: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit\"\ntime=\"2021-08-02T21:30:22Z\" level=warning msg=\"failed, retrying in 1s ... (3/3). Error: Error initializing source docker://internetsystemsconsortium/bind9:9.11: Error reading manifest 9.11 in docker.io/internetsystemsconsortium/bind9: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit\"\nError: Error initializing source docker://internetsystemsconsortium/bind9:9.11: Error reading manifest 9.11 in docker.io/internetsystemsconsortium/bind9: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit\n", "stderr_lines": ["Trying to pull docker.io/internetsystemsconsortium/bind9:9.11...", "time=\"2021-08-02T21:29:16Z\" level=warning msg=\"failed, retrying in 1s ... (1/3). Error: Error initializing source docker://internetsystemsconsortium/bind9:9.11: Error reading manifest 9.11 in docker.io/internetsystemsconsortium/bind9: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit\"", "time=\"2021-08-02T21:29:49Z\" level=warning msg=\"failed, retrying in 1s ... (2/3). Error: Error initializing source docker://internetsystemsconsortium/bind9:9.11: Error reading manifest 9.11 in docker.io/internetsystemsconsortium/bind9: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit\"", "time=\"2021-08-02T21:30:22Z\" level=warning msg=\"failed, retrying in 1s ... (3/3). Error: Error initializing source docker://internetsystemsconsortium/bind9:9.11: Error reading manifest 9.11 in docker.io/internetsystemsconsortium/bind9: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit\"", "Error: Error initializing source docker://internetsystemsconsortium/bind9:9.11: Error reading manifest 9.11 in docker.io/internetsystemsconsortium/bind9: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit"], "stdout": "", "stdout_lines": []}
PLAY RECAP *********************************************************************
hypervisor-01 : ok=19 changed=3 unreachable=0 failed=0 skipped=6 rescued=0 ignored=1
localhost : ok=220 changed=103 unreachable=0 failed=1 skipped=54 rescued=0 ignored=0
time="2021-08-02T17:30:55-04:00" level=error msg="unable to cleanup network for container 0cce952948ba14f61c57a6ccc3a7533a02f409404b9ac704d8f585eede9666ed: \"error tearing down CNI namespace configuration for container 0cce952948ba14f61c57a6ccc3a7533a02f409404b9ac704d8f585eede9666ed: Error while removing pod from CNI network \\\"podman\\\": running [/usr/sbin/iptables -t nat -D POSTROUTING -s 10.88.0.39 -j CNI-a12114ad2239e4242b4847a6 -m comment --comment name: \\\"podman\\\" id: \\\"0cce952948ba14f61c57a6ccc3a7533a02f409404b9ac704d8f585eede9666ed\\\" --wait]: exit status 2: iptables v1.8.4 (nf_tables): Chain 'CNI-a12114ad2239e4242b4847a6' does not exist\\nTry `iptables -h' or 'iptables --help' for more information.\\n\""
(ara.sh) ==> Executing ara.sh
(ara.sh) ==> The pipelne_id is rke-libvirt-1-0-1-
It seems that when cleaning the pod network it tries to use IPtables instead of firewalld to manage the rules.
[root@nyctea ~]# cat /etc/cni/net.d/eks-service-01-bridge.conflist
{
"cniVersion": "0.4.0",
"name": "eks-service-01-bridge",
"plugins": [
{
"type": "bridge",
"bridge": "cni-podman3",
"isGateway": true,
"ipMasq": true,
"mtu": 1442,
"hairpinMode": true,
"ipam": {
"type": "host-local",
"routes": [
{
"dst": "0.0.0.0/0"
}
],
"ranges": [
[
{
"subnet": "10.89.2.0/24",
"gateway": "10.89.2.1"
}
]
]
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
},
{
"type": "firewall",
"backend": ""
},
{
"type": "tuning"
}
]
In this case, the firewall network plugin backend should be firewalld.
This:
{
"type": "firewall",
"backend": ""
},
To:
{
"type": "firewall",
"backend": "firewalld"
}
Solutions:
- Make sure we don't install or enable firewalld anywhere not enabled by default.
- We install firewalld
- We enable firewalld
- Update the CNI configuration file with the correct backend in case firewalld is enabled.
This error causes an unhandled exception in Ansible that blocks the logs gathering so the periodic jobs output data is empty.
[root@nyctea ~]# podman network inspect podman |grep -C1 firewall
{
"type": "firewall"
},
From what I can see the driver can be nftables, iptables, or firewalld.
Some references: