ip6tables rules (silently) not added due to invalid index
chschu opened this issue · comments
If firewall.EnsureRules(...) detects an existing rule, the corresponding ruleCounters entry is not incremented. When a docker container disconnects from a network later on, firewall.RemoveRules decrements the counter, making it negative.
go-iptables reports an error, which unfortunately gets ignored there:
Line 111 in 38c8e82
Because errors are ignored, some rules are not added correctly, which led to seemingly random connectivity issues when I recreated some docker containers.
When propagating the error on that line, ipv6nat terminates with messages like these:
2019/04/11 01:34:54 running [/sbin/ip6tables -t nat -I POSTROUTING -1 -s fd00:ca7e:d09e::4 -d fd00:ca7e:d09e::4 -p tcp -m tcp --dport 143 -j MASQUERADE --wait]: exit status 2: ip6tables v1.6.0: unknown option "-1"
There also seems to be some scenario where the index becomes too large:
2019/04/11 01:37:47 running [/sbin/ip6tables -t filter -I DOCKER 7 -d fd00:ca7e:d09e::2 ! -i br-bc678bb0c0ae -o br-bc678bb0c0ae -p tcp -m tcp --dport 443 -j ACCEPT --wait]: exit status 1: ip6tables: Index of insertion too big.
Maybe fw.ipt.Append(...) could be used to append the non-prepend rules. This would eliminate the need for ruleCounters. Keeping the counters in sync if there are pre-existing rules seems pretty complicated.
Hi @chschu, great find and thanks for taking the time to dive into the cause of this! I believe I once used Append to add the non-prepend rules, but people with existing IPv6 firewall rules were running into problems: the appended rules were added after the pre-existing rules and the prepended rules before. In order to fix this I created this running count of rules created by ipv6nat to always insert rules above the pre-existing rules. It has always been a bit of a hacky fix and this issue proves it isn't robust enough.
Unfortunately, using Append will be a step back, reintroducing the issues other people were having with pre-existing rules.
Thanks for your response, @robbertkl. You're right, ipv6nat should probably never add any rules after any pre-existing rule.
Incrementing the ruleCounters entry when inserting a rule seems to be fine, but it must not be decremented when deleting a pre-existing rule.
A possible solution: Mirroring the added/removed rules for each chain as keys in a map[string]bool and using the length of that map to determine the insertion point for iptables. The rule itself cannot be used as a map key because it contains a slice, which is mutable. Some kind of string concatenation of the rule spec should be fine.
I've implemented your solution and from what I've tested it works like a charm. I've created a new release (v0.3.5) and I'm running it in production now to see if any problems arise. Docker images for this version will be done building soon.
Thanks again! Let me know if the issue is resolved on your end as well.
Works like a charm for me. Thanks @robbertkl for the quick fix and release!