Issue with swarm mode using docker_gwbridge bridge

[quentindavid]

Hi ! Thanks a lot for your work, very surprising now in end-2020 we still have to fight in order to have in a consistent behaviour between IPV4 and IPV6 within docker..

So, I followed your documentation, and it works well for containers sitting in docker, I can see the rules added in the debug mode. Now I try to use it with swarm mode, so I enabled IPV6 on docker_gwbridge :

docker network create \
 --ipv6 \
 --subnet 172.25.0.0/16 \
 --gateway 172.25.0.1 \
 --gateway fdd0:4cab:5070:357f::1 \
 --subnet fdd0:4cab:5070:357f::/64 \
 --opt com.docker.network.bridge.name=docker_gwbridge \
 --opt com.docker.network.bridge.enable_icc=true \
 --opt com.docker.network.bridge.enable_ip_forwarding=true \
 --opt com.docker.network.bridge.enable_ip_masquerade=true \
 docker_gwbridge

And then I launched the container :

docker run -d --name Ipv6nat --privileged --network host --restart unless-stopped -v /var/run/docker.sock:/var/run/docker.sock:ro -v /lib/modules:/lib/modules:ro robbertkl/ipv6nat -cleanup -debug -retry

Now I can see the container is able to see the network docker_gwbridge because I see this in the container logs:

2020/11/24 16:07:12 rule added: -t filter -A FORWARD 11 -o docker_gwbridge -j DOCKER
2020/11/24 16:07:12 rule added: -t filter -A FORWARD 12 -o docker_gwbridge -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
2020/11/24 16:07:12 rule added: -t filter -A FORWARD 13 -i docker_gwbridge ! -o docker_gwbridge -j ACCEPT
2020/11/24 16:07:13 rule added: -t filter -A FORWARD 14 -i docker_gwbridge -o docker_gwbridge -j ACCEPT
2020/11/24 16:07:13 rule added: -t nat -A DOCKER 1 -i docker_gwbridge -j RETURN
2020/11/24 16:07:13 rule added: -t nat -A POSTROUTING 1 -s fdd0:4cab:5070:357f::/64 ! -o docker_gwbridge -j MASQUERADE
2020/11/24 16:07:13 rule added: -t nat -A POSTROUTING 1 -o docker_gwbridge -m addrtype --dst-type LOCAL -j MASQUERADE
2020/11/24 16:07:13 rule added: -t filter -A DOCKER-ISOLATION-STAGE-2 1 -o docker_gwbridge -j DROP
2020/11/24 16:07:13 rule added: -t filter -A DOCKER-ISOLATION-STAGE-1 1 -i docker_gwbridge ! -o docker_gwbridge -j DOCKER-ISOLATION-STAGE-2

But I cannot see any automatic rule like I see for simple containers. If I do manually this (fdd0:4cab:5070:357f::5 is the IP of a container within a swarm stack):

ip6tables -t filter -A DOCKER -d fdd0:4cab:5070:357f::5 ! -i docker0 -o docker0 -p tcp -m tcp --dport 443 -j ACCEPT
ip6tables -t nat -A DOCKER -d 0/0 -p tcp -m tcp --dport 443 -j DNAT --to-destination [fdd0:4cab:5070:357f::5]:443 ! -i docker0

then it works...so it seems there is an issue in order to detected the container when it's within a swarm.

Do someone know if I miss something ? IPV6 seems to be working fine, I am able to ping6 external IP from my containers, including those within the swarm.

Thanks again a lot !!

[Robbert Klarenbeek]

Hi @quentindavid, sorry you're running into issues. I don't use swarm mode myself. Perhaps @Chrislevi, who wrote the swarm mode instructions in the README, knows something about this?

[quentindavid]

Hi @robbertkl , thanks for answering this fast !
The containers, within a swarm or not, are containers, and they are connected to docker_gwbridge, so as I understood it should work fine.
I tried to read the code but I'm not sure to understand how it works. Is it able to detected both containers connected to docker0 and docker_gwbridge as well ?

Thanks again :)

[Robbert Klarenbeek]

Yeah, the code is listening for Docker events, so it should see any container being connected to a network.

[quentindavid]

Ok then I don't understand why it's missing the containers connected within swarm. Maybe @Chrislevi is able to help ?

[indrajeetray]

Hi @Chrislevi,

I am facing the same issue too.
Earlier I was using bridge network with IPv6 and I could see ipv6 NAT working. I have a setup where I expect UDP packets to come from outside my system and it is forwarded to the container with the source IP unchanged.

I am trying to migrade away from docker default bridge to docker_gwbridge that we get when we have swarm network. When I am connected to docker_gwbridge (swarm network)... I see the packets reaching the VM and then forwarded to the container using IPv4 address of the docker_gwbridge in the VM instead of the source IPv6 address retained in the packet.

Any pointers to debug this.

These are the options in the docker_gwbridge network (as seen from docker network inspect docker_gwbridge command)

        "Options": {
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_forwarding": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.name": "docker_gwbridge"
        },

docker logs for ipv6nat in debug mode has following output:

2021/03/19 03:46:15 docker-ipv6nat is running in debug mode
2021/03/19 03:46:15 rule added: -t filter -A DOCKER-ISOLATION-STAGE-1 1 -j RETURN
2021/03/19 03:46:15 rule added: -t filter -A DOCKER-ISOLATION-STAGE-2 1 -j RETURN
2021/03/19 03:46:15 rule added: -t filter -A FORWARD 1 -j DOCKER-USER
2021/03/19 03:46:15 rule added: -t nat -A DOCKER 1 -i docker_gwbridge -j RETURN
2021/03/19 03:46:15 rule added: -t filter -A DOCKER-ISOLATION-STAGE-2 1 -o docker_gwbridge -j DROP
2021/03/19 03:46:15 rule added: -t filter -A DOCKER-ISOLATION-STAGE-1 1 -i docker_gwbridge ! -o docker_gwbridge -j DOCKER-ISOLATION-STAGE-2
2021/03/19 03:46:15 rule added: -t nat -A DOCKER 1 -i docker0 -j RETURN
2021/03/19 03:46:15 rule added: -t filter -A DOCKER-ISOLATION-STAGE-2 1 -o docker0 -j DROP
2021/03/19 03:46:15 rule added: -t filter -A DOCKER-ISOLATION-STAGE-1 1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2