ipv6nat stop working after docker update
celogeek opened this issue · comments
Hi,
I have got this error since my docker upgrade:
ipv6nat_1 | Try `iptables -h' or 'iptables --help' for more information.
ipv6nat_1 | 2019/05/25 21:20:31 running [/sbin/iptables -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER --wait]: exit status 2: iptables v1.6.2: Couldn't load target `DOCKER':No such file or directory
My docker version:
docker version
Client:
Version: 18.09.6
API version: 1.39
Go version: go1.10.8
Git commit: 481bc77
Built: Sat May 4 02:36:01 2019
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 18.09.6
API version: 1.39 (minimum version 1.12)
Go version: go1.10.8
Git commit: 481bc77
Built: Sat May 4 01:59:36 2019
OS/Arch: linux/amd64
Experimental: false
My nat table:
sudo iptables -L -t nat
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
DOCKER all -- anywhere anywhere ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE *** (all my rules here)
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
DOCKER all -- anywhere !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
Chain DOCKER (2 references)
target prot opt source destination
RETURN all -- anywhere anywhere
RETURN all -- anywhere anywhere
DNAT *** (all my ipv4 dnat here)
# Warning: iptables-legacy tables present, use iptables-legacy to see them
I have disabled ipv6 for now. Can I do something to fix this?
thanks
I'm not sure where those errors come from, ipv6nat runs ip6tables, not iptables. It seems something else is going on here.
I can take a look, but please give me your run command (or docker-compose section) for ipv6nat and output of both iptables-save and ip6tables-save.
iptables-save:
# Generated by xtables-save v1.8.2 on Sun May 26 11:16:23 2019
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o br-5c0b4b729dfe -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-5c0b4b729dfe -j DOCKER
-A FORWARD -i br-5c0b4b729dfe ! -o br-5c0b4b729dfe -j ACCEPT
-A FORWARD -i br-5c0b4b729dfe -o br-5c0b4b729dfe -j ACCEPT
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 10.20.0.4/32 ! -i br-5c0b4b729dfe -o br-5c0b4b729dfe -p tcp -m tcp --dport 993 -j ACCEPT
-A DOCKER -d 10.20.0.5/32 ! -i br-5c0b4b729dfe -o br-5c0b4b729dfe -p tcp -m tcp --dport 22 -j ACCEPT
-A DOCKER -d 10.20.0.6/32 ! -i br-5c0b4b729dfe -o br-5c0b4b729dfe -p tcp -m tcp --dport 587 -j ACCEPT
-A DOCKER -d 10.20.0.6/32 ! -i br-5c0b4b729dfe -o br-5c0b4b729dfe -p tcp -m tcp --dport 465 -j ACCEPT
-A DOCKER -d 10.20.0.6/32 ! -i br-5c0b4b729dfe -o br-5c0b4b729dfe -p tcp -m tcp --dport 25 -j ACCEPT
-A DOCKER -d 10.20.0.8/32 ! -i br-5c0b4b729dfe -o br-5c0b4b729dfe -p tcp -m tcp --dport 22 -j ACCEPT
-A DOCKER -d 10.20.0.10/32 ! -i br-5c0b4b729dfe -o br-5c0b4b729dfe -p tcp -m tcp --dport 51415 -j ACCEPT
-A DOCKER -d 10.20.0.12/32 ! -i br-5c0b4b729dfe -o br-5c0b4b729dfe -p tcp -m tcp --dport 22 -j ACCEPT
-A DOCKER -d 10.20.0.20/32 ! -i br-5c0b4b729dfe -o br-5c0b4b729dfe -p tcp -m tcp --dport 443 -j ACCEPT
-A DOCKER -d 10.20.0.20/32 ! -i br-5c0b4b729dfe -o br-5c0b4b729dfe -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i br-5c0b4b729dfe ! -o br-5c0b4b729dfe -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o br-5c0b4b729dfe -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Sun May 26 11:16:23 2019
# Generated by xtables-save v1.8.2 on Sun May 26 11:16:23 2019
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 10.20.0.0/24 ! -o br-5c0b4b729dfe -j MASQUERADE
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 10.20.0.4/32 -d 10.20.0.4/32 -p tcp -m tcp --dport 993 -j MASQUERADE
-A POSTROUTING -s 10.20.0.5/32 -d 10.20.0.5/32 -p tcp -m tcp --dport 22 -j MASQUERADE
-A POSTROUTING -s 10.20.0.6/32 -d 10.20.0.6/32 -p tcp -m tcp --dport 587 -j MASQUERADE
-A POSTROUTING -s 10.20.0.6/32 -d 10.20.0.6/32 -p tcp -m tcp --dport 465 -j MASQUERADE
-A POSTROUTING -s 10.20.0.6/32 -d 10.20.0.6/32 -p tcp -m tcp --dport 25 -j MASQUERADE
-A POSTROUTING -s 10.20.0.8/32 -d 10.20.0.8/32 -p tcp -m tcp --dport 22 -j MASQUERADE
-A POSTROUTING -s 10.20.0.10/32 -d 10.20.0.10/32 -p tcp -m tcp --dport 51415 -j MASQUERADE
-A POSTROUTING -s 10.20.0.12/32 -d 10.20.0.12/32 -p tcp -m tcp --dport 22 -j MASQUERADE
-A POSTROUTING -s 10.20.0.20/32 -d 10.20.0.20/32 -p tcp -m tcp --dport 443 -j MASQUERADE
-A POSTROUTING -s 10.20.0.20/32 -d 10.20.0.20/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A DOCKER -i br-5c0b4b729dfe -j RETURN
-A DOCKER -i docker0 -j RETURN
-A DOCKER ! -i br-5c0b4b729dfe -p tcp -m tcp --dport 993 -j DNAT --to-destination 10.20.0.4:993
-A DOCKER ! -i br-5c0b4b729dfe -p tcp -m tcp --dport 10025 -j DNAT --to-destination 10.20.0.5:22
-A DOCKER ! -i br-5c0b4b729dfe -p tcp -m tcp --dport 587 -j DNAT --to-destination 10.20.0.6:587
-A DOCKER ! -i br-5c0b4b729dfe -p tcp -m tcp --dport 465 -j DNAT --to-destination 10.20.0.6:465
-A DOCKER ! -i br-5c0b4b729dfe -p tcp -m tcp --dport 25 -j DNAT --to-destination 10.20.0.6:25
-A DOCKER ! -i br-5c0b4b729dfe -p tcp -m tcp --dport 10023 -j DNAT --to-destination 10.20.0.8:22
-A DOCKER ! -i br-5c0b4b729dfe -p tcp -m tcp --dport 51415 -j DNAT --to-destination 10.20.0.10:51415
-A DOCKER ! -i br-5c0b4b729dfe -p tcp -m tcp --dport 10024 -j DNAT --to-destination 10.20.0.12:22
-A DOCKER ! -i br-5c0b4b729dfe -p tcp -m tcp --dport 443 -j DNAT --to-destination 10.20.0.20:443
-A DOCKER ! -i br-5c0b4b729dfe -p tcp -m tcp --dport 80 -j DNAT --to-destination 10.20.0.20:80
COMMIT
# Completed on Sun May 26 11:16:23 2019
ip6tables-save:
# Generated by xtables-save v1.8.2 on Sun May 26 11:16:52 2019
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT
# Completed on Sun May 26 11:16:52 2019
# Generated by xtables-save v1.8.2 on Sun May 26 11:16:52 2019
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT
# Completed on Sun May 26 11:16:52 2019
docker-compose.yml:
networks:
default:
driver: bridge
enable_ipv6: true
ipam:
driver: default
config:
- subnet: 10.20.0.0/24
- subnet: fd00:dead:beef::/48
services:
ipv6nat:
image: robbertkl/ipv6nat
command: ["-retry", "-cleanup"]
restart: always
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- /lib/modules:/lib/modules:ro
network_mode: host
cap_add:
- NET_ADMIN
- SYS_MODULE
And this output is with ipv6nat running? Or did it exit at this time? I'm asking because you're using --cleanup which removes all the rules on exit.
Can you do a docker-compose up -d --force-recreate ipv6nat and after that send me the docker logs?
I have removed the command section and run the recreate:
ipv6nat_1 | 2019/05/26 11:25:07 running [/sbin/iptables -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER --wait]: exit status 2: iptables v1.6.2: Couldn't load target `DOCKER':No such file or directory
ipv6nat_1 |
ipv6nat_1 | Try `iptables -h' or 'iptables --help' for more information.
The ipv6nat wasn't running because it fails at the start with that log.
OK, it craps out trying to detect hairpin mode in the IPv4 firewall. Could you try:
docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock:ro -v /lib/modules:/lib/modules:ro --net=host --cap-add=NET_ADMIN --cap-add=SYS_MODULE --entrypoint iptables-save robbertkl/ipv6nat
sudo docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock:ro -v /lib/modules:/lib/modules:ro --net=host --cap-add=NET_ADMIN --cap-add=SYS_MODULE --entrypoint iptables-save robbertkl/ipv6nat
# Generated by iptables-save v1.6.2 on Sun May 26 12:14:56 2019
*nat
:PREROUTING ACCEPT [8392:426047]
:INPUT ACCEPT [2635:96038]
:OUTPUT ACCEPT [3987:318688]
:POSTROUTING ACCEPT [9744:648697]
COMMIT
# Completed on Sun May 26 12:14:56 2019
Can you try the same but with --privileged instead?
docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock:ro -v /lib/modules:/lib/modules:ro --net=host --privileged --entrypoint iptables-save robbertkl/ipv6nat
The output is empty.
When I run docker-ipv6nat again:
/ # ./docker-ipv6nat
2019/05/26 15:50:19 running [/sbin/iptables -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER --wait]: exit status 2: iptables v1.6.2: Couldn't load target `DOCKER':No such file or directory
So when you're running iptables-save from within a privileged, net=host ipv6nat container it shows no IPv4 rules, but when you run iptables-save on the host it does show IPv4 rules?
I've just tried the same on a test host running Docker 19.03.0-beta3, and I still see the IPv4 rules from within the container, so no way to reproduce it on my end.
Sorry I run the ip6tables command.
Here the output:
# Generated by iptables-save v1.6.2 on Sun May 26 16:20:42 2019
*nat
:PREROUTING ACCEPT [3580:160229]
:INPUT ACCEPT [562:19464]
:OUTPUT ACCEPT [792:64734]
:POSTROUTING ACCEPT [1927:130179]
COMMIT
# Completed on Sun May 26 16:20:42 2019
# Generated by iptables-save v1.6.2 on Sun May 26 16:20:42 2019
*filter
:INPUT ACCEPT [3180:338688]
:FORWARD ACCEPT [50554:39971344]
:OUTPUT ACCEPT [3327:460405]
COMMIT
# Completed on Sun May 26 16:20:42 2019
Sorry, I'm a bit confused, this output is different than before again. Can you please post all 3 outputs:
iptables-save(on the host)docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock:ro -v /lib/modules:/lib/modules:ro --net=host --cap-add=NET_ADMIN --cap-add=SYS_MODULE --entrypoint iptables-save robbertkl/ipv6natdocker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock:ro -v /lib/modules:/lib/modules:ro --net=host --privileged --entrypoint iptables-save robbertkl/ipv6nat
All commands use iptables, not ip6tables. The problem is your ipv6nat instance seems not to see the IPv4 rules, even though they're available on the host. I cannot reproduce this myself, however.
Despite my provider give me only a /128 block, it seems I can have a /64 block anyway.
They are not limiting the ipv6 assignation on my /64 block.
I have reconfigured my docker to handle ipv6 block /80.
All my containers have a public IP address now, and it seems to connect to the host with ipv6 on a container port (SMTP for example) redirect properly but in a weird way.
HOST ipv6 -> Host docker ipv4 -> Container IPV4
Also, an issue is that they are no ipv6 isolation of the container. If I have a port listing (80) on a container but I don't open it on the host (don't have the "ports"), I can't reach it with ipv4, but I can with ipv6.
Also, ports redirect doesn't work well: ports: 2222:22, give ipv4: 2222, ipv6: 22
The ipv6 nat is far easier to configure.
Here the output I have (sorry for the change):
1.
# Generated by xtables-save v1.8.2 on Sun May 26 16:29:11 2019
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o br-a040ef404950 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-a040ef404950 -j DOCKER
-A FORWARD -i br-a040ef404950 ! -o br-a040ef404950 -j ACCEPT
-A FORWARD -i br-a040ef404950 -o br-a040ef404950 -j ACCEPT
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 10.20.0.27/32 ! -i br-a040ef404950 -o br-a040ef404950 -p tcp -m tcp --dport 51415 -j ACCEPT
-A DOCKER -d 10.20.0.28/32 ! -i br-a040ef404950 -o br-a040ef404950 -p tcp -m tcp --dport 22 -j ACCEPT
-A DOCKER -d 10.20.0.15/32 ! -i br-a040ef404950 -o br-a040ef404950 -p tcp -m tcp --dport 22 -j ACCEPT
-A DOCKER -d 10.20.0.19/32 ! -i br-a040ef404950 -o br-a040ef404950 -p tcp -m tcp --dport 587 -j ACCEPT
-A DOCKER -d 10.20.0.19/32 ! -i br-a040ef404950 -o br-a040ef404950 -p tcp -m tcp --dport 465 -j ACCEPT
-A DOCKER -d 10.20.0.19/32 ! -i br-a040ef404950 -o br-a040ef404950 -p tcp -m tcp --dport 25 -j ACCEPT
-A DOCKER -d 10.20.0.21/32 ! -i br-a040ef404950 -o br-a040ef404950 -p tcp -m tcp --dport 22 -j ACCEPT
-A DOCKER -d 10.20.0.18/32 ! -i br-a040ef404950 -o br-a040ef404950 -p tcp -m tcp --dport 993 -j ACCEPT
-A DOCKER -d 10.20.0.10/32 ! -i br-a040ef404950 -o br-a040ef404950 -p tcp -m tcp --dport 443 -j ACCEPT
-A DOCKER -d 10.20.0.10/32 ! -i br-a040ef404950 -o br-a040ef404950 -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i br-a040ef404950 ! -o br-a040ef404950 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o br-a040ef404950 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Sun May 26 16:29:11 2019
# Generated by xtables-save v1.8.2 on Sun May 26 16:29:11 2019
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 10.20.0.0/24 ! -o br-a040ef404950 -j MASQUERADE
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 10.20.0.27/32 -d 10.20.0.27/32 -p tcp -m tcp --dport 51415 -j MASQUERADE
-A POSTROUTING -s 10.20.0.28/32 -d 10.20.0.28/32 -p tcp -m tcp --dport 22 -j MASQUERADE
-A POSTROUTING -s 10.20.0.15/32 -d 10.20.0.15/32 -p tcp -m tcp --dport 22 -j MASQUERADE
-A POSTROUTING -s 10.20.0.19/32 -d 10.20.0.19/32 -p tcp -m tcp --dport 587 -j MASQUERADE
-A POSTROUTING -s 10.20.0.19/32 -d 10.20.0.19/32 -p tcp -m tcp --dport 465 -j MASQUERADE
-A POSTROUTING -s 10.20.0.19/32 -d 10.20.0.19/32 -p tcp -m tcp --dport 25 -j MASQUERADE
-A POSTROUTING -s 10.20.0.21/32 -d 10.20.0.21/32 -p tcp -m tcp --dport 22 -j MASQUERADE
-A POSTROUTING -s 10.20.0.18/32 -d 10.20.0.18/32 -p tcp -m tcp --dport 993 -j MASQUERADE
-A POSTROUTING -s 10.20.0.10/32 -d 10.20.0.10/32 -p tcp -m tcp --dport 443 -j MASQUERADE
-A POSTROUTING -s 10.20.0.10/32 -d 10.20.0.10/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A DOCKER -i br-a040ef404950 -j RETURN
-A DOCKER -i docker0 -j RETURN
-A DOCKER ! -i br-a040ef404950 -p tcp -m tcp --dport 51415 -j DNAT --to-destination 10.20.0.27:51415
-A DOCKER ! -i br-a040ef404950 -p tcp -m tcp --dport 10025 -j DNAT --to-destination 10.20.0.28:22
-A DOCKER ! -i br-a040ef404950 -p tcp -m tcp --dport 10023 -j DNAT --to-destination 10.20.0.15:22
-A DOCKER ! -i br-a040ef404950 -p tcp -m tcp --dport 587 -j DNAT --to-destination 10.20.0.19:587
-A DOCKER ! -i br-a040ef404950 -p tcp -m tcp --dport 465 -j DNAT --to-destination 10.20.0.19:465
-A DOCKER ! -i br-a040ef404950 -p tcp -m tcp --dport 25 -j DNAT --to-destination 10.20.0.19:25
-A DOCKER ! -i br-a040ef404950 -p tcp -m tcp --dport 10024 -j DNAT --to-destination 10.20.0.21:22
-A DOCKER ! -i br-a040ef404950 -p tcp -m tcp --dport 993 -j DNAT --to-destination 10.20.0.18:993
-A DOCKER ! -i br-a040ef404950 -p tcp -m tcp --dport 443 -j DNAT --to-destination 10.20.0.10:443
-A DOCKER ! -i br-a040ef404950 -p tcp -m tcp --dport 80 -j DNAT --to-destination 10.20.0.10:80
COMMIT
# Completed on Sun May 26 16:29:11 2019
# Generated by iptables-save v1.6.2 on Sun May 26 16:29:40 2019
*nat
:PREROUTING ACCEPT [4531:202007]
:INPUT ACCEPT [731:25374]
:OUTPUT ACCEPT [990:81507]
:POSTROUTING ACCEPT [2369:161300]
COMMIT
# Completed on Sun May 26 16:29:40 2019
# Generated by iptables-save v1.6.2 on Sun May 26 16:29:40 2019
*filter
:INPUT ACCEPT [3977:421198]
:FORWARD ACCEPT [85777:73490326]
:OUTPUT ACCEPT [4113:547670]
COMMIT
# Completed on Sun May 26 16:29:40 2019
# Generated by iptables-save v1.6.2 on Sun May 26 16:29:56 2019
*nat
:PREROUTING ACCEPT [4554:202959]
:INPUT ACCEPT [735:25502]
:OUTPUT ACCEPT [992:81688]
:POSTROUTING ACCEPT [2374:161665]
COMMIT
# Completed on Sun May 26 16:29:56 2019
# Generated by iptables-save v1.6.2 on Sun May 26 16:29:56 2019
*filter
:INPUT ACCEPT [3995:422953]
:FORWARD ACCEPT [86786:74429505]
:OUTPUT ACCEPT [4131:549825]
COMMIT
# Completed on Sun May 26 16:29:56 2019
well, ipv6 sucks with docker. I will disable ipv6 on my host. It is not really useful.
If that evolves, I may reenable it.
Or I may look out for an alternative.
Yeah, it does suck big time. However, I've been running it with ipv6nat for years without any issues. Not sure what your issue is; your container not being able to reach the (IPv4) firewall rules seems to be the cause of ipv6nat crashing, but why this is happening I don't know. Are you by any chance using SELinux? I can't reproduce it myself, so it's hard to debug.
I'm on debian/buster, the latest kernel on this system.
I also have the latest (non-beta) version of docker/docker-compose.
It seems to hide iptables rules indeed inside the container which cause the issue here.
Yeah, I'm curious to know what's causing it. Even on 19.03.0-beta3 I could not reproduce it, so it must not be a Docker thing. Perhaps it's something kernel-related.
Can you test it with debian/buster ?
Maybe it would be possible and safer to run your companion on the host directly? Something like a service that starts with docker?
The kernel:
Linux ns363135 4.19.0-5-amd64 #1 SMP Debian 4.19.37-3 (2019-05-15) x86_64 GNU/Linux
Running on the host is a great idea, that should work since it should be able to access the firewall there.
I'll try to do some tests with buster / later kernels soon.
OK @celogeek, just did 2 clean Debian VM installs (1 stretch @ 4.9.0-9, 1 buster @ 4.19.0-5) and installed Docker stable (18.09.6).
Then tried reading iptables from within a (privileged, net=host) container on both:
- on stretch it worked
- on buster it did not (no rules)
It seems the problem is buster using nftables (see https://wiki.debian.org/nftables). I managed to get it working by switching to legacy version by running:
update-alternatives --set iptables /usr/sbin/iptables-legacy
update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
update-alternatives --set arptables /usr/sbin/arptables-legacy
update-alternatives --set ebtables /usr/sbin/ebtables-legacy
reboot
After the reboot, I can see the rules from inside the container again.
Nice catch!
Did you try directly on the host? without a container?
Yes, you can either switch to legacy iptables like above, or run ipv6nat directly on the host. On the host it will use the iptables/ip6tables commands that are linked to the special nftables compatibility versions.
This same approach would also work from within the container (auto translating iptables to nftables), so I'm working on a fix for the Docker image to support this. I just need to figure out how to reliably detect which is used. After I'm done with this fix, you should be able to run it again as a container, without any changes.
Hi @celogeek , just released a new version of the docker images with support for nftables, so it should work again with your previous setup, unchanged. Could you test this?
Yes of course.
I will need to reactivate ipv6 on my server.
I do that as soon as possible.
I guess you have try in debian/buster/nfptables condition ?
Yep, tried it and as far as I could see it starts just fine.
test done
it doesn't work.
here the error I got:
ipv6nat_1 | 2019/05/28 18:50:57 running [/usr/local/bin/ip6tables -t filter -I DOCKER 1 -d fd00:dead:beef::2 ! -i br-f016aef740c8 -o br-f016aef740c8 -p tcp -m tcp --dport 22 -j ACCEPT --wait]: exit status 1: iptables: Invalid argument. Run `dmesg' for more information.
in dmesg:
[ 557.022382] x_tables: ip6_tables: tcp match: only valid for protocol 6
Maybe used ip6tables-nft instead?
Yeah, I'm seeing this same error now. Seems like a bug in the compat scripts, because we're obviously using protocol 6. Doing an ip6tables-translate did not generate an error.
You're right about ip6tables-nft, this is what I wanted to use, however I'm using Alpine Linux as a base image and their iptables is running behind.
Wait, this is great, there's a Alpine ip6tables 1.8.2-r1 package in edge now, built just YESTERDAY
I'll go check it out.
I got a bit further by using 1.8.1-r1 and ip6tables-nft, but now I ran into an issue with go-iptables, I reported it here: coreos/go-iptables#59
There's not really a workaround with the current version, so I'll fix the issue to get it working. Should be done soon, I'll let you know. I'll reopen this issue.
@celogeek Could you please test again with the latest release? Thanks!
yes sure. testing it right now.
It works !!! congrat.
Everything is back to normal now.
I will take a vps with ipv6 connectivity and play with real ipv6 config (no nat).
It seems to be not so easy to do a firewall with docker / ufw / ipv6.
This would be a good challenge ;)
A very great companion for docker would be something like this tools, that disable incoming connection and redirect port to the one inside but with real ipv6 ip addr.
As I said before if I have a real ipv6 on a docker, it is accessible directly without following the rules of dockers (port open/close or redirected).
Good to hear it's working! I'll close the issue now.