ip6tables error

Question

ip6tables error

bstegmaier75 opened this issue 5 years ago · comments

I tried to use ipv6nat with docker-ce 18.09.6~~3-0~~debian-buster on a current Debian Testing.
A sample nginx container is running on an IPv6 enabled network configured as fd01::/64.

When I start ipv6nat (using docker-compose up) I get the following error:

ipv6nat_1 | 2019/05/29 19:52:44 running [/usr/local/bin/ip6tables -t filter -I DOCKER 1 -d fd01::2 ! -i br-4895d4b90f94 -o br-4895d4b90f94 -p tcp -m tcp --dport 80 -j ACCEPT --wait]: exit status 1: iptables: Invalid argument. Run `dmesg' for more information.

dmesg shows:

[613582.357457] x_tables: ip6_tables: tcp match: only valid for protocol 6

I am running nftables on the host machine.
Stock docker IPv4 NAT works fine.

Robbert Klarenbeek · Answer 1 · Thu May 30 2019 04:05:31 GMT+0800 (China Standard Time)

Hi @bstegmaier75, thanks for reporting. I'm aware of the issue (see discussion at #33) and working on a fix. I'll leave this bug open until it's fixed.

bstegmaier75 · Answer 2 · Thu May 30 2019 04:07:58 GMT+0800 (China Standard Time)

OK, sorry, didn't read #33 until the end...
Anyway... thanks!

Robbert Klarenbeek · Answer 3 · Thu May 30 2019 04:08:56 GMT+0800 (China Standard Time)

Yeah, sorry, #33 was closed prematurely 😉

Robbert Klarenbeek · Answer 4 · Thu May 30 2019 08:58:26 GMT+0800 (China Standard Time)

@bstegmaier75 could you pull the latest image and try again? Thanks!

bstegmaier75 · Answer 5 · Thu May 30 2019 15:53:54 GMT+0800 (China Standard Time)

Seems to work fine now, thanks!

It doesn't work to connect to the webserver, but from what I see IPv6 config of my local interfaces seems to be completely broken (I tried various things)... I have to fix that first.

That's what has been generated for a single port 8000:80 mapping:

table ip6 filter {
chain INPUT {
type filter hook input priority 0; policy accept;
}

chain FORWARD {
type filter hook forward priority 0; policy accept;
counter packets 8 bytes 448 jump DOCKER-ISOLATION
oifname "br-abc0cbaf4052" counter packets 0 bytes 0 jump DOCKER
oifname "br-abc0cbaf4052" ct state related,established counter packets 0 bytes 0 accept
iifname "br-abc0cbaf4052" oifname != "br-abc0cbaf4052" counter packets 0 bytes 0 accept
iifname "br-abc0cbaf4052" oifname "br-abc0cbaf4052" counter packets 0 bytes 0 accept
}

chain OUTPUT {
type filter hook output priority 0; policy accept;
}

chain DOCKER {
iifname != "br-abc0cbaf4052" oifname "br-abc0cbaf4052" meta l4proto tcp ip6 daddr fd01::2 tcp dport 80 counter packets 0 bytes 0 accept
}

chain DOCKER-ISOLATION {
counter packets 8 bytes 448 return
}
}
table ip6 nat {
chain PREROUTING {
type nat hook prerouting priority -100; policy accept;
fib daddr type local counter packets 0 bytes 0 jump DOCKER
}

chain INPUT {
type nat hook input priority 100; policy accept;
}

chain POSTROUTING {
type nat hook postrouting priority 100; policy accept;
oifname != "br-abc0cbaf4052" ip6 saddr fd01::/64 counter packets 0 bytes 0 masquerade
meta l4proto tcp ip6 saddr fd01::2 ip6 daddr fd01::2 tcp dport 80 counter packets 0 bytes 0 masquerade
}

chain OUTPUT {
type nat hook output priority -100; policy accept;
fib daddr type local counter packets 0 bytes 0 jump DOCKER
}

chain DOCKER {
meta l4proto tcp tcp dport 8000 counter packets 0 bytes 0 dnat to [fd01::2]:80
}
}

bstegmaier75 · Answer 6 · Thu May 30 2019 16:25:06 GMT+0800 (China Standard Time)

Reboot seems to have fixed IPv6 config for all interfaces.
But, still can't connect to webserver via IPv6 address.
I can ping the host with its IPv6, so in general IPv6 setup should be OK.

I am no expert on nftables and I still have find out yet how to setup the NAT rules manually, but there is one thing I am not sure about:

In several places there is something like that:

  meta l4proto tcp ip6 saddr fd01::2 ip6 daddr fd01::2 tcp dport 80 counter packets 0 bytes 0 masquerade

Interesting is "... meta l4proto tcp ip6 saddr fd01::2 ..."

In DOCKER chain there is

  meta l4proto tcp tcp dport 8000 counter packets 10 bytes 840 dnat to [fd01::2]:80

Here there is "... meta l4proto tcp tcp dport 8000 ..." instead of tcp ip6

Might that be a problem?

Robbert Klarenbeek · Answer 7 · Thu May 30 2019 16:57:46 GMT+0800 (China Standard Time)

I don't think that's the problem, those are matches and they're split up a bit differently:

meta l4proto tcp ip6 saddr fd01::2 ip6 daddr fd01::2 tcp dport 80 counter packets 0 bytes 0 masquerade:

meta l4proto tcp
ip6 saddr fd01::2
ip6 daddr fd01::2
tcp dport 80
...

meta l4proto tcp tcp dport 8000 counter packets 10 bytes 840 dnat to [fd01::2]:80:

meta l4proto tcp
tcp dport 8000
`...

What's the output of ip6tables-save? (this translates the underlying nft rules back to iptables format)

bstegmaier75 · Answer 8 · Thu May 30 2019 20:09:25 GMT+0800 (China Standard Time)

ip6tables-save on host:

# Generated by xtables-save v1.8.2 on Thu May 30 13:56:18 2019
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o br-86624d2f14b9 -j DOCKER
-A FORWARD -o br-86624d2f14b9 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i br-86624d2f14b9 ! -o br-86624d2f14b9 -j ACCEPT
-A FORWARD -i br-86624d2f14b9 -o br-86624d2f14b9 -j ACCEPT
-A DOCKER -d fd01::2/128 ! -i br-86624d2f14b9 -o br-86624d2f14b9 -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER-ISOLATION -j RETURN
COMMIT
# Completed on Thu May 30 13:56:18 2019
# Generated by xtables-save v1.8.2 on Thu May 30 13:56:18 2019
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s fd01::/64 ! -o br-86624d2f14b9 -j MASQUERADE
-A POSTROUTING -s fd01::2/128 -d fd01::2/128 -p tcp -m tcp --dport 80 -j MASQUERADE
-A OUTPUT -m addrtype --dst-type LOCAL -j DOCKER
-A DOCKER -p tcp -m tcp --dport 8000 -j DNAT --to-destination [fd01::2]:80
COMMIT
# Completed on Thu May 30 13:56:18 2019

Looks pretty much the same as the IPv4 stuff, as far as I can see.

Route should be also OK:

fd01::/64                      [::]                       U    256 3     0 br-86624d2f14b9
fd01::/128                     [::]                       Un   0   3     0 br-86624d2f14b9
fd01::1/128                    [::]                       Un   0   4     0 br-86624d2f14b9

Robbert Klarenbeek · Answer 9 · Thu May 30 2019 20:18:57 GMT+0800 (China Standard Time)

Yeah, this looks OK. Are you sure the problem is with docker-ipv6nat or could it be something else in your IPv6 setup? Was everything working properly before you upgraded to buster?

bstegmaier75 · Answer 10 · Thu May 30 2019 20:28:15 GMT+0800 (China Standard Time)

No, not sure that it is with ipv6nat.
I (tried to) set it up for the first time yesterday.

I checked the counters of the various rules and they don't change when I try to connect, so it seems as if I don't even reach the host and I probably have some problem with my IPv6 setup.

Just to be sure: I have my router configured to announce a fd00:: ULA to my local network and the docker network to be fd01::. That's how it is supposed to be?

Robbert Klarenbeek · Answer 11 · Thu May 30 2019 20:31:23 GMT+0800 (China Standard Time)

I'm not familiar with a setup like that, but it should be good as long as the ULA prefixes are different. I (and most users) have a publicly available IPv6 address on a server and use a ULA for the Docker with NAT.

Robbert Klarenbeek · Answer 12 · Thu May 30 2019 20:50:56 GMT+0800 (China Standard Time)

Try to get basic IPv6 connectivity working first, can you ping6 from the host?

I'll close the issue now, since the original error was resolved. I've got confirmation in #33 that everything is working now with buster/nftables. Feel free to continue discussion here to get your setup working.

bstegmaier75 · Answer 13 · Thu May 30 2019 21:09:19 GMT+0800 (China Standard Time)

Silly me.
Some leftover firewall rules from my before-docker setup caught traffic from docker network to host.
Removed them and everything is working as it should.

Thanks for the support!

Robbert Klarenbeek · Answer 14 · Thu May 30 2019 21:10:36 GMT+0800 (China Standard Time)

Great! Have a good day.