Bug: failure to renew port forwarding shuts down the container
dmitry-t7ko opened this issue Β· comments
Is this urgent?
Yes
Host OS
endeavouros
CPU arch
x86_64
VPN service provider
ProtonVPN
What are you using to run the container
docker-compose
What is the version of Gluetun
Running version latest built on 2024-05-09T14:36:40.530Z (commit ce642a6)
What's the problem π€
Marked urgent, because container shuts itself down.
Occasionally with proton vpn port forwarding setup (on either wireguard or openvpn), I see that container shut down after the following error:
ERROR port forwarding loop crashed: stopping previous service: blocking previous port in firewall: removing allowed port 51876 on interface tun0: command failed: "iptables --delete INPUT -i tun0 -p tcp --dport 51876 -j ACCEPT": iptables: Bad rule (does a matching rule exist in that chain?).: exit status 1
This issue is intermittent, and usually appears within 5 minutes of starting container. All the other services connected via gluetun are rendered inaccessible.
My understanding is that port forwarding is being renewed on proton server every 60 seconds, and it that fails gluetun double-frees the iptables rule for the port being forwarded.
Note: I've been seeing other failures to connect to proton, which usually look like this:
gluetun | 2024-05-09T22:26:51Z INFO [dns] downloading DNS over TLS cryptographic files
gluetun | 2024-05-09T22:27:06Z WARN [dns] cannot update files: Get "https://www.internic.net/domain/named.root": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
gluetun | 2024-05-09T22:27:06Z INFO [dns] attempting restart in 10s
and repeating. It might be related to me pulling a lot of data in short amount of time, and proton throttling me (even though running wireguard config on host - without port forwarding - seems to be ok). I'm not sure what to do with it yet, but I can try and provide more info if needed.
Share your logs (at least 10 lines)
gluetun | ========================================
gluetun | ========================================
gluetun | =============== gluetun ================
gluetun | ========================================
gluetun | =========== Made with β€ by ============
gluetun | ======= https://github.com/qdm12 =======
gluetun | ========================================
gluetun | ========================================
gluetun |
gluetun | Running version latest built on 2024-05-09T14:36:40.530Z (commit ce642a6)
gluetun |
gluetun | π§ Need help? https://github.com/qdm12/gluetun/discussions/new
gluetun | π Bug? https://github.com/qdm12/gluetun/issues/new
gluetun | β¨ New feature? https://github.com/qdm12/gluetun/issues/new
gluetun | β Discussion? https://github.com/qdm12/gluetun/discussions/new
gluetun | π» Email? quentin.mcgaw@gmail.com
gluetun | π° Help me? https://www.paypal.me/qmcgaw https://github.com/sponsors/qdm12
gluetun | 2024-05-09T21:48:09Z INFO [routing] default route found: interface eth0, gateway 192.168.55.1, assigned IP 192.168.55.10 and family v4
gluetun | 2024-05-09T21:48:09Z INFO [routing] local ethernet link found: eth0
gluetun | 2024-05-09T21:48:09Z INFO [routing] local ipnet found: 192.168.55.0/24
gluetun | 2024-05-09T21:48:09Z INFO [firewall] enabling...
gluetun | 2024-05-09T21:48:09Z INFO [firewall] enabled successfully
gluetun | 2024-05-09T21:48:10Z INFO [storage] merging by most recent 19425 hardcoded servers and 19425 servers read from /gluetun/servers.json
gluetun | 2024-05-09T21:48:10Z INFO Alpine version: 3.19.1
gluetun | 2024-05-09T21:48:10Z INFO OpenVPN 2.5 version: 2.5.8
gluetun | 2024-05-09T21:48:10Z INFO OpenVPN 2.6 version: 2.6.8
gluetun | 2024-05-09T21:48:10Z INFO Unbound version: 1.20.0
gluetun | 2024-05-09T21:48:10Z INFO IPtables version: v1.8.10
gluetun | 2024-05-09T21:48:10Z INFO Settings summary:
gluetun | βββ VPN settings:
gluetun | | βββ VPN provider settings:
gluetun | | | βββ Name: custom
gluetun | | | βββ Server selection settings:
gluetun | | | | βββ VPN type: wireguard
gluetun | | | | βββ Target IP address: [REDACTED]
gluetun | | | | βββ Wireguard selection settings:
gluetun | | | | βββ Endpoint IP address: [REDACTED]
gluetun | | | | βββ Endpoint port: 51820
gluetun | | | | βββ Server public key: [REDACTED]
gluetun | | | βββ Automatic port forwarding settings:
gluetun | | | βββ Redirection listening port: disabled
gluetun | | | βββ Use code for provider: protonvpn
gluetun | | | βββ Forwarded port file path: /tmp/gluetun/forwarded_port
gluetun | | βββ Wireguard settings:
gluetun | | βββ Private key: wAe...3o=
gluetun | | βββ Interface addresses:
gluetun | | | βββ 10.2.0.2/32
gluetun | | βββ Allowed IPs:
gluetun | | | βββ 0.0.0.0/0
gluetun | | | βββ ::/0
gluetun | | βββ Network interface: tun0
gluetun | | βββ MTU: 1400
gluetun | βββ DNS settings:
gluetun | | βββ Keep existing nameserver(s): no
gluetun | | βββ DNS server address to use: 127.0.0.1
gluetun | | βββ DNS over TLS settings:
gluetun | | βββ Enabled: yes
gluetun | | βββ Update period: every 24h0m0s
gluetun | | βββ Unbound settings:
gluetun | | | βββ Authoritative servers:
gluetun | | | | βββ cloudflare
gluetun | | | βββ Caching: yes
gluetun | | | βββ IPv6: no
gluetun | | | βββ Verbosity level: 1
gluetun | | | βββ Verbosity details level: 0
gluetun | | | βββ Validation log level: 0
gluetun | | | βββ System user: root
gluetun | | | βββ Allowed networks:
gluetun | | | βββ 0.0.0.0/0
gluetun | | | βββ ::/0
gluetun | | βββ DNS filtering settings:
gluetun | | βββ Block malicious: yes
gluetun | | βββ Block ads: no
gluetun | | βββ Block surveillance: no
gluetun | | βββ Blocked IP networks:
gluetun | | βββ 127.0.0.1/8
gluetun | | βββ 10.0.0.0/8
gluetun | | βββ 172.16.0.0/12
gluetun | | βββ 192.168.0.0/16
gluetun | | βββ 169.254.0.0/16
gluetun | | βββ ::1/128
gluetun | | βββ fc00::/7
gluetun | | βββ fe80::/10
gluetun | | βββ ::ffff:127.0.0.1/104
gluetun | | βββ ::ffff:10.0.0.0/104
gluetun | | βββ ::ffff:169.254.0.0/112
gluetun | | βββ ::ffff:172.16.0.0/108
gluetun | | βββ ::ffff:192.168.0.0/112
gluetun | βββ Firewall settings:
gluetun | | βββ Enabled: yes
gluetun | βββ Log settings:
gluetun | | βββ Log level: info
gluetun | βββ Health settings:
gluetun | | βββ Server listening address: 127.0.0.1:9999
gluetun | | βββ Target address: cloudflare.com:443
gluetun | | βββ Duration to wait after success: 2m0s
gluetun | | βββ Read header timeout: 100ms
gluetun | | βββ Read timeout: 500ms
gluetun | | βββ VPN wait durations:
gluetun | | βββ Initial duration: 1m0s
gluetun | | βββ Additional duration: 30s
gluetun | βββ Shadowsocks server settings:
gluetun | | βββ Enabled: no
gluetun | βββ HTTP proxy settings:
gluetun | | βββ Enabled: no
gluetun | βββ Control server settings:
gluetun | | βββ Listening address: :8000
gluetun | | βββ Logging: yes
gluetun | βββ OS Alpine settings:
gluetun | | βββ Process UID: 1000
gluetun | | βββ Process GID: 1000
gluetun | βββ Public IP settings:
gluetun | | βββ Fetching: every 12h0m0s
gluetun | | βββ IP file path: /tmp/gluetun/ip
gluetun | | βββ Public IP data API: ipinfo
gluetun | βββ Version settings:
gluetun | βββ Enabled: yes
gluetun | 2024-05-09T21:48:10Z INFO [routing] default route found: interface eth0, gateway 192.168.55.1, assigned IP 192.168.55.10 and family v4
gluetun | 2024-05-09T21:48:10Z INFO [routing] adding route for 0.0.0.0/0
gluetun | 2024-05-09T21:48:10Z INFO [firewall] setting allowed subnets...
gluetun | 2024-05-09T21:48:10Z INFO [routing] default route found: interface eth0, gateway 192.168.55.1, assigned IP 192.168.55.10 and family v4
gluetun | 2024-05-09T21:48:10Z INFO [dns] using plaintext DNS at address 1.1.1.1
gluetun | 2024-05-09T21:48:10Z INFO [http server] http server listening on [::]:8000
gluetun | 2024-05-09T21:48:10Z INFO [firewall] allowing VPN connection...
gluetun | 2024-05-09T21:48:10Z INFO [healthcheck] listening on 127.0.0.1:9999
gluetun | 2024-05-09T21:48:10Z INFO [wireguard] Using available kernelspace implementation
gluetun | 2024-05-09T21:48:10Z INFO [wireguard] Connecting to 149.88.20.129:51820
gluetun | 2024-05-09T21:48:10Z INFO [wireguard] Wireguard setup is complete. Note Wireguard is a silent protocol and it may or may not work, without giving any error message. Typically i/o timeout errors indicate the Wireguard connection is not working.
gluetun | 2024-05-09T21:48:10Z INFO [dns] downloading DNS over TLS cryptographic files
gluetun | 2024-05-09T21:48:11Z INFO [healthcheck] healthy!
gluetun | 2024-05-09T21:48:11Z INFO [dns] downloading hostnames and IP block lists
gluetun | 2024-05-09T21:48:14Z INFO [dns] init module 0: validator
gluetun | 2024-05-09T21:48:14Z INFO [dns] init module 1: iterator
gluetun | 2024-05-09T21:48:14Z INFO [dns] start of service (unbound 1.20.0).
gluetun | 2024-05-09T21:48:15Z INFO [dns] generate keytag query _ta-4a5c-4f66. NULL IN
gluetun | 2024-05-09T21:48:15Z INFO [dns] generate keytag query _ta-4a5c-4f66. NULL IN
gluetun | 2024-05-09T21:48:15Z INFO [dns] ready
gluetun | 2024-05-09T21:48:15Z INFO [ip getter] Public IP address is [REDACTED]
gluetun | 2024-05-09T21:48:15Z INFO [vpn] You are running on the bleeding edge of latest!
gluetun | 2024-05-09T21:48:15Z INFO [port forwarding] starting
gluetun | 2024-05-09T21:48:15Z INFO [port forwarding] gateway external IPv4 address is [REDACTED]
gluetun | 2024-05-09T21:48:16Z INFO [port forwarding] port forwarded is 51876
gluetun | 2024-05-09T21:48:16Z INFO [firewall] setting allowed input port 51876 through interface tun0...
gluetun | 2024-05-09T21:48:16Z INFO [port forwarding] writing port file /tmp/gluetun/forwarded_port
gluetun | 2024-05-09T21:50:41Z INFO [healthcheck] healthy!
gluetun | 2024-05-09T21:55:12Z INFO [firewall] removing allowed port 51876...
gluetun | 2024-05-09T21:55:12Z ERROR [port forwarding] adding port mapping: executing remote procedure call: connection timeout: failed attempts: read udp 10.2.0.2:44773->10.2.0.1:5351: i/o timeout (tries 1, 2, 3, 4, 5, 6, 7, 8, 9)
gluetun | 2024-05-09T21:55:12Z INFO [port forwarding] stopping
gluetun | 2024-05-09T21:55:12Z INFO [firewall] removing allowed port 51876...
gluetun | 2024-05-09T21:55:12Z ERROR port forwarding loop crashed: stopping previous service: blocking previous port in firewall: removing allowed port 51876 on interface tun0: command failed: "iptables --delete INPUT -i tun0 -p tcp --dport 51876 -j ACCEPT": iptables: Bad rule (does a matching rule exist in that chain?).: exit status 1
gluetun | 2024-05-09T21:55:12Z INFO dns ticker: terminated β
gluetun | 2024-05-09T21:55:12Z INFO updater ticker: terminated β
gluetun | 2024-05-09T21:55:12Z INFO http server: terminated β
gluetun | 2024-05-09T21:55:12Z INFO control: terminated β
gluetun | 2024-05-09T21:55:12Z INFO updater: terminated β
gluetun | 2024-05-09T21:55:12Z INFO tickers: terminated β
gluetun | 2024-05-09T21:55:12Z INFO HTTP health server: terminated β
gluetun | 2024-05-09T21:55:13Z WARN vpn: goroutine shutdown timed out: after 1s β
gluetun | 2024-05-09T21:55:13Z INFO shadowsocks proxy: terminated β
gluetun | 2024-05-09T21:55:13Z INFO http proxy: terminated β
gluetun | 2024-05-09T21:55:13Z INFO unbound: terminated β
gluetun | 2024-05-09T21:55:13Z INFO other: terminated β
gluetun | 2024-05-09T21:55:13Z INFO [routing] routing cleanup...
gluetun | 2024-05-09T21:55:13Z INFO [routing] default route found: interface eth0, gateway 192.168.55.1, assigned IP 192.168.55.10 and family v4
gluetun | 2024-05-09T21:55:13Z INFO [routing] deleting route for 0.0.0.0/0
gluetun | 2024-05-09T21:55:13Z ERROR ordered shutdown timed out: vpn: goroutine shutdown timed out: after 1s
gluetun | 2024-05-09T21:55:13Z INFO Shutdown successful
Share your configuration
services:
gluetun:
image: qmcgaw/gluetun
container_name: gluetun
cap_add:
- NET_ADMIN
devices:
- /dev/net/tun:/dev/net/tun
ports:
- 8100:8100
- 8110:8080
- "3333:3333"
- "3334:3334/tcp"
- "3334:3334/udp"
# - 8888:8888/tcp # HTTP proxy
# - 8388:8388/tcp # Shadowsocks
# - 8388:8388/udp # Shadowsocks
environment:
- VPN_TYPE=wireguard
- VPN_SERVICE_PROVIDER=custom
- VPN_ENDPOINT_IP=[REDACTED]
- VPN_ENDPOINT_PORT=[REDACTED]
- WIREGUARD_PUBLIC_KEY=[REDACTED]
- WIREGUARD_PRIVATE_KEY=[REDACTED]
- WIREGUARD_ADDRESSES=[REDACTED]
- VPN_PORT_FORWARDING=on
- VPN_PORT_FORWARDING_PROVIDER=protonvpn
@qdm12 is more or less the only maintainer of this project and works on it in his free time.
Please:
- do not ask for updates, be patient
- π the issue to show your support instead of commenting
@qdm12 usually checks issues at least once a week, if this is a new urgent bug,
revert to an older tagged container image
I also have a problem with protonvpn with port forwarding "on", running the latest version of gluetun.
Here are my logs:
| βββ Interface addresses:
| | βββ 10.2.0.2/32
| βββ Allowed IPs:
| | βββ 0.0.0.0/0
| | βββ ::/0
| βββ Network interface: tun0
| βββ MTU: 1400
βββ DNS settings:
| βββ Keep existing nameserver(s): no
| βββ DNS server address to use: 127.0.0.1
| βββ DNS over TLS settings:
| βββ Enabled: yes
| βββ Update period: every 24h0m0s
| βββ Unbound settings:
| | βββ Authoritative servers:
| | | βββ cloudflare
| | βββ Caching: yes
| | βββ IPv6: no
| | βββ Verbosity level: 1
| | βββ Verbosity details level: 0
| | βββ Validation log level: 0
| | βββ System user: root
| | βββ Allowed networks:
| | βββ 0.0.0.0/0
| | βββ ::/0
| βββ DNS filtering settings:
| βββ Block malicious: yes
| βββ Block ads: no
| βββ Block surveillance: no
| βββ Blocked IP networks:
| βββ 127.0.0.1/8
| βββ 10.0.0.0/8
| βββ 172.16.0.0/12
| βββ 192.168.0.0/16
| βββ 169.254.0.0/16
| βββ ::1/128
| βββ fc00::/7
| βββ fe80::/10
| βββ ::ffff:127.0.0.1/104
| βββ ::ffff:10.0.0.0/104
| βββ ::ffff:169.254.0.0/112
| βββ ::ffff:172.16.0.0/108
| βββ ::ffff:192.168.0.0/112
βββ Firewall settings:
| βββ Enabled: yes
βββ Log settings:
| βββ Log level: info
βββ Health settings:
| βββ Server listening address: 127.0.0.1:9999
| βββ Target address: cloudflare.com:443
| βββ Duration to wait after success: 5s
| βββ Read header timeout: 100ms
| βββ Read timeout: 500ms
| βββ VPN wait durations:
| βββ Initial duration: 6s
| βββ Additional duration: 5s
βββ Shadowsocks server settings:
| βββ Enabled: no
βββ HTTP proxy settings:
| βββ Enabled: no
βββ Control server settings:
| βββ Listening address: :8000
| βββ Logging: yes
βββ OS Alpine settings:
| βββ Process UID: 1000
| βββ Process GID: 1000
βββ Public IP settings:
| βββ Fetching: every 12h0m0s
| βββ IP file path: /tmp/gluetun/ip
| βββ Public IP data API: ipinfo
βββ Version settings:
βββ Enabled: yes
2024-05-10T02:24:48Z INFO [routing] default route found: interface eth0, gateway 172.28.0.1, assigned IP 172.28.0.5 and family v4
2024-05-10T02:24:48Z INFO [routing] adding route for 0.0.0.0/0
2024-05-10T02:24:48Z INFO [firewall] setting allowed subnets...
2024-05-10T02:24:48Z INFO [routing] default route found: interface eth0, gateway 172.28.0.1, assigned IP 172.28.0.5 and family v4
2024-05-10T02:24:48Z INFO TUN device is not available: open /dev/net/tun: no such file or directory; creating it...
2024-05-10T02:24:48Z INFO [dns] using plaintext DNS at address 1.1.1.1
2024-05-10T02:24:48Z INFO [http server] http server listening on [::]:8000
2024-05-10T02:24:48Z INFO [firewall] allowing VPN connection...
2024-05-10T02:24:48Z INFO [healthcheck] listening on 127.0.0.1:9999
2024-05-10T02:24:48Z INFO [wireguard] Using available kernelspace implementation
2024-05-10T02:24:48Z INFO [wireguard] Connecting to [REDACTED]:51820
2024-05-10T02:24:48Z INFO [wireguard] Wireguard setup is complete. Note Wireguard is a silent protocol and it may or may not work, without giving any error message. Typically i/o timeout errors indicate the Wireguard connection is not working.
2024-05-10T02:24:48Z INFO [dns] downloading DNS over TLS cryptographic files
2024-05-10T02:24:56Z INFO [healthcheck] program has been unhealthy for 6s: restarting VPN
2024-05-10T02:24:56Z INFO [healthcheck] π See https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md
2024-05-10T02:24:56Z INFO [healthcheck] DO NOT OPEN AN ISSUE UNLESS YOU READ AND TRIED EACH POSSIBLE SOLUTION
2024-05-10T02:24:56Z INFO [vpn] stopping
2024-05-10T02:24:56Z ERROR [vpn] cannot get version information: Get "https://api.github.com/repos/qdm12/gluetun/commits": context canceled
2024-05-10T02:24:56Z INFO [port forwarding] starting
2024-05-10T02:24:56Z ERROR [vpn] port forwarding for the first time: getting external IPv4 address: executing remote procedure call: writing to connection: write udp 172.28.0.5:60139->10.2.0.1:5351: write: operation not permitted
2024-05-10T02:24:56Z ERROR [ip getter] Get "https://ipinfo.io/": dial tcp: lookup ipinfo.io on 1.1.1.1:53: write udp 172.28.0.5:47948->1.1.1.1:53: write: operation not permitted - retrying in 5s
2024-05-10T02:24:56Z INFO [vpn] starting
2024-05-10T02:24:56Z INFO [firewall] allowing VPN connection...
2024-05-10T02:24:56Z INFO [wireguard] Using available kernelspace implementation
2024-05-10T02:24:56Z INFO [wireguard] Connecting to 169.150.204.33:51820
2024-05-10T02:24:56Z INFO [wireguard] Wireguard setup is complete. Note Wireguard is a silent protocol and it may or may not work, without giving any error message. Typically i/o timeout errors indicate the Wireguard connection is not working.
2024-05-10T02:25:03Z WARN [dns] cannot update files: Get "https://www.internic.net/domain/named.root": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-05-10T02:25:03Z INFO [dns] attempting restart in 10s
Downgrading to an earlier version didn't help.
I also saw that yesterday actually, it's due to nf_tables misbehaving (I reported the bug to the netfilter project) which is now the default backend for iptables since the upgrade to Alpine 3.19. I have a local fix which prefers using the legacy version of iptables (not using nf_tables). I'll push it later today, in the meantime use :v3.38
Actually the fix was pushed yesterday in commit ce642a6 so just re-pull the latest image and it should be fixed. I'll close this assuming this is resolved.
Closed issues are NOT monitored, so commenting here is likely to be not seen.
If you think this is still unresolved and have more information to bring, please create another issue.
This is an automated comment setup because @qdm12 is the sole maintainer of this project
which became too popular to monitor issues closed.