🐛 2024.1.3 broke stability of connection

Question

🐛 2024.1.3 broke stability of connection

nwpr opened this issue 5 months ago · comments

Describe the bug
cloudflared 2024.1.3 immediately broke the stability of the connection, after updating to this version.
The tunnel is flapping up and down (getting health emails from cf, anything from healthy/down/degraded), and the connection is nearly unusable.

After the bug occurred, I spawned another cloudflared replica using cloudflare/cloudflared:2024.1.2, and the connection using 2024.1.2 is working perfectly as soon as the replica gets active.
Also tried to completely recreate the container with 2024.1.3 from scratch. Same result.

To Reproduce
Steps to reproduce the behavior:

Run the docker container with 2024.1.3.
The tunnel is flapping up and down, and the connection is nearly unusable.

If it's an issue with Cloudflare Tunnel:

Tunnel ID : beba9dda-8fc8-44fe-8ebd-91d2382ea9c1 (and also e1a33431-1889-483f-9677-235a944fbc07)
cloudflared config: tunnel --no-autoupdate run --token !!!REDACTED!!!

Expected behavior
The connection should be stable - as in 2024.1.2.

Environment and versions

OS: Docker version 24.0.5, build ced0996, on Photon 3.0
Architecture: AMD64
Version: 2024.1.3

Logs and errors

{"log":"2024-01-19T03:16:36Z INF Starting tunnel tunnelID=beba9dda-8fc8-44fe-8ebd-91d2382ea9c1\n","stream":"stderr","time":"2024-01-19T03:16:36.4766452Z"}
{"log":"2024-01-19T03:16:36Z INF Version 2024.1.3\n","stream":"stderr","time":"2024-01-19T03:16:36.476838735Z"}
{"log":"2024-01-19T03:16:36Z INF GOOS: linux, GOVersion: go1.21.5-devel-cf, GoArch: amd64\n","stream":"stderr","time":"2024-01-19T03:16:36.476854232Z"}
{"log":"2024-01-19T03:16:36Z INF Settings: map[no-autoupdate:true token:*****]\n","stream":"stderr","time":"2024-01-19T03:16:36.476862583Z"}
{"log":"2024-01-19T03:16:36Z INF Generated Connector ID: cebd7dad-8b3b-46c0-9415-d393fa939fe8\n","stream":"stderr","time":"2024-01-19T03:16:36.477691586Z"}
{"log":"2024-01-19T03:16:36Z INF Initial protocol quic\n","stream":"stderr","time":"2024-01-19T03:16:36.483743283Z"}
{"log":"2024-01-19T03:16:36Z INF ICMP proxy will use 10.10.11.22 as source for IPv4\n","stream":"stderr","time":"2024-01-19T03:16:36.617806634Z"}
{"log":"2024-01-19T03:16:36Z INF ICMP proxy will use :: as source for IPv6\n","stream":"stderr","time":"2024-01-19T03:16:36.617855951Z"}
{"log":"2024-01-19T03:16:36Z INF Starting metrics server on 127.0.0.1:39883/metrics\n","stream":"stderr","time":"2024-01-19T03:16:36.617870284Z"}
{"log":"2024-01-19T03:16:36Z INF Registered tunnel connection connIndex=0 connection=1b0f4ac1-7c7c-4cf7-bbf8-cb574f1cf2b4 event=0 ip=198.41.200.23 location=vie02 protocol=quic\n","stream":"stderr","time":"2024-01-19T03:16:36.765062503Z"}
{"log":"2024-01-19T03:16:37Z INF Updated to new configuration config=\ !!!REDACTED!!!
{"log":"2024-01-19T03:16:37Z INF Registered tunnel connection connIndex=1 connection=8815410e-fb08-4c82-bcae-a8661879fecb event=0 ip=198.41.192.227 location=bud01 protocol=quic\n","stream":"stderr","time":"2024-01-19T03:16:37.429034383Z"}
{"log":"2024-01-19T03:16:38Z INF Registered tunnel connection connIndex=2 connection=16045cf5-092a-4a51-9417-c5450d1db326 event=0 ip=198.41.200.233 location=vie06 protocol=quic\n","stream":"stderr","time":"2024-01-19T03:16:38.34721523Z"}
{"log":"2024-01-19T03:16:38Z WRN Failed to serve quic connection error=\"already connected to this server, trying another address\" connIndex=3 event=0 ip=198.41.192.67\n","stream":"stderr","time":"2024-01-19T03:16:38.855514152Z"}
{"log":"2024-01-19T03:16:38Z WRN Unable to establish connection. error=\"already connected to this server, trying another address\" connIndex=3 event=0 ip=198.41.192.67\n","stream":"stderr","time":"2024-01-19T03:16:38.855542427Z"}
{"log":"2024-01-19T03:16:38Z INF Retrying connection in up to 2s connIndex=3 event=0 ip=198.41.192.67\n","stream":"stderr","time":"2024-01-19T03:16:38.855549618Z"}
{"log":"2024-01-19T03:16:39Z WRN Connection terminated error=\"already connected to this server, trying another address\" connIndex=3\n","stream":"stderr","time":"2024-01-19T03:16:39.765900231Z"}
{"log":"2024-01-19T03:16:58Z INF Registered tunnel connection connIndex=3 connection=8171c1db-b573-47da-9232-b31f8e93b4df event=0 ip=198.41.192.167 location=bud01 protocol=quic\n","stream":"stderr","time":"2024-01-19T03:16:58.121757991Z"}
{"log":"2024-01-19T03:23:30Z ERR  error=\"dial tcp 192.168.1.184:7000: i/o timeout\" event=2 flowID=d10c9ba7-0935-486a-a805-5dc11968aa66 originService=warp-routing\n","stream":"stderr","time":"2024-01-19T03:23:30.224587436Z"}
{"log":"2024-01-19T03:23:30Z ERR Request failed error=\"dial tcp 192.168.1.184:7000: i/o timeout\" connIndex=0 dest=192.168.1.184:7000 event=0 ip=198.41.200.23 type=tcp\n","stream":"stderr","time":"2024-01-19T03:23:30.224624848Z"}
{"log":"2024-01-19T03:25:13Z ERR  error=\"dial tcp 10.3.61.101:7: connect: no route to host\" event=2 flowID=ec9e628a-7fa9-4dbb-b1fd-a1d666815732 originService=warp-routing\n","stream":"stderr","time":"2024-01-19T03:25:13.698187324Z"}
{"log":"2024-01-19T03:25:13Z ERR Request failed error=\"dial tcp 10.3.61.101:7: connect: no route to host\" connIndex=0 dest=10.3.61.101:7 event=0 ip=198.41.200.23 type=tcp\n","stream":"stderr","time":"2024-01-19T03:25:13.698234953Z"}
{"log":"2024-01-19T03:25:13Z ERR  error=\"dial tcp 10.3.61.101:7: connect: no route to host\" event=2 flowID=3c1dedcd-da43-42b0-96f6-2b1c20276cdd originService=warp-routing\n","stream":"stderr","time":"2024-01-19T03:25:13.716578842Z"}
{"log":"2024-01-19T03:25:13Z ERR Request failed error=\"dial tcp 10.3.61.101:7: connect: no route to host\" connIndex=0 dest=10.3.61.101:7 event=0 ip=198.41.200.23 type=tcp\n","stream":"stderr","time":"2024-01-19T03:25:13.716615699Z"}
{"log":"2024-01-19T03:25:13Z ERR  error=\"dial tcp 10.3.61.101:7: connect: no route to host\" event=2 flowID=daff7315-308e-4098-8082-ddcdfb841b7f originService=warp-routing\n","stream":"stderr","time":"2024-01-19T03:25:13.716623199Z"}
{"log":"2024-01-19T03:25:13Z ERR Request failed error=\"dial tcp 10.3.61.101:7: connect: no route to host\" connIndex=0 dest=10.3.61.101:7 event=0 ip=198.41.200.23 type=tcp\n","stream":"stderr","time":"2024-01-19T03:25:13.716647406Z"}
{"log":"2024-01-19T03:25:13Z ERR  error=\"dial tcp 10.3.61.102:7: connect: no route to host\" event=2 flowID=588a6e97-93b9-4722-8b36-c50b9714b5d2 originService=warp-routing\n","stream":"stderr","time":"2024-01-19T03:25:13.716716124Z"}
{"log":"2024-01-19T03:25:13Z ERR Request failed error=\"dial tcp 10.3.61.102:7: connect: no route to host\" connIndex=0 dest=10.3.61.102:7 event=0 ip=198.41.200.23 type=tcp\n","stream":"stderr","time":"2024-01-19T03:25:13.716741485Z"}
{"log":"2024-01-19T03:25:14Z ERR  error=\"dial tcp 10.3.61.102:7: connect: no route to host\" event=2 flowID=f683c3c0-d35d-4a54-81c1-de74bd1c7f20 originService=warp-routing\n","stream":"stderr","time":"2024-01-19T03:25:14.468065035Z"}
{"log":"2024-01-19T03:25:14Z ERR Request failed error=\"dial tcp 10.3.61.102:7: connect: no route to host\" connIndex=2 dest=10.3.61.102:7 event=0 ip=198.41.200.233 type=tcp\n","stream":"stderr","time":"2024-01-19T03:25:14.468103053Z"}
{"log":"2024-01-19T03:25:21Z ERR  error=\"dial tcp 192.168.1.145:7000: i/o timeout\" event=2 flowID=d36808be-bab6-442d-be2c-82d04f0f2e5b originService=warp-routing\n","stream":"stderr","time":"2024-01-19T03:25:21.633286155Z"}
{"log":"2024-01-19T03:25:21Z ERR  error=\"dial tcp 192.168.1.145:7000: i/o timeout\" event=2 flowID=a52feb42-5a0e-4b0f-9cbc-372ea4c62a79 originService=warp-routing\n","stream":"stderr","time":"2024-01-19T03:25:21.633325926Z"}
{"log":"2024-01-19T03:25:21Z ERR Request failed error=\"dial tcp 192.168.1.145:7000: i/o timeout\" connIndex=2 dest=192.168.1.145:7000 event=0 ip=198.41.200.233 type=tcp\n","stream":"stderr","time":"2024-01-19T03:25:21.633333004Z"}
{"log":"2024-01-19T03:25:21Z ERR Request failed error=\"dial tcp 192.168.1.145:7000: i/o timeout\" connIndex=2 dest=192.168.1.145:7000 event=0 ip=198.41.200.233 type=tcp\n","stream":"stderr","time":"2024-01-19T03:25:21.633361384Z"}
{"log":"2024-01-19T03:25:26Z ERR  error=\"dial tcp 10.3.61.101:7: i/o timeout\" event=2 flowID=a8a28382-b1b9-41d1-aae7-ab5c776eace2 originService=warp-routing\n","stream":"stderr","time":"2024-01-19T03:25:26.145254704Z"}
{"log":"2024-01-19T03:25:26Z ERR Request failed error=\"dial tcp 10.3.61.101:7: i/o timeout\" connIndex=1 dest=10.3.61.101:7 event=0 ip=198.41.192.227 type=tcp\n","stream":"stderr","time":"2024-01-19T03:25:26.145295165Z"}
{"log":"2024-01-19T03:25:43Z ERR  error=\"dial tcp 192.168.1.145:7000: i/o timeout\" event=2 flowID=0302aed6-9b92-4348-bfd6-4591846e86da originService=warp-routing\n","stream":"stderr","time":"2024-01-19T03:25:43.921363282Z"}
{"log":"2024-01-19T03:25:43Z ERR Request failed error=\"dial tcp 192.168.1.145:7000: i/o timeout\" connIndex=1 dest=192.168.1.145:7000 event=0 ip=198.41.192.227 type=tcp\n","stream":"stderr","time":"2024-01-19T03:25:43.921404085Z"}
{"log":"2024-01-19T03:25:43Z ERR  error=\"dial tcp 192.168.1.145:7000: i/o timeout\" event=2 flowID=fec06e57-01ef-440c-8307-2e0086ab3bc8 originService=warp-routing\n","stream":"stderr","time":"2024-01-19T03:25:43.923629424Z"}
{"log":"2024-01-19T03:25:43Z ERR Request failed error=\"dial tcp 192.168.1.145:7000: i/o timeout\" connIndex=1 dest=192.168.1.145:7000 event=0 ip=198.41.192.227 type=tcp\n","stream":"stderr","time":"2024-01-19T03:25:43.923734906Z"}
{"log":"2024-01-19T03:25:43Z ERR  error=\"dial tcp 192.168.1.145:7000: i/o timeout\" event=2 flowID=8984873a-22a5-4ddf-8fe9-30729d68f719 originService=warp-routing\n","stream":"stderr","time":"2024-01-19T03:25:43.991858642Z"}
{"log":"2024-01-19T03:25:43Z ERR Request failed error=\"dial tcp 192.168.1.145:7000: i/o timeout\" connIndex=1 dest=192.168.1.145:7000 event=0 ip=198.41.192.227 type=tcp\n","stream":"stderr","time":"2024-01-19T03:25:43.991897591Z"}
{"log":"2024-01-19T03:25:48Z ERR  error=\"dial tcp 10.3.61.102:7: i/o timeout\" event=2 flowID=1626748f-34aa-4b43-b127-a4323c0d8555 originService=warp-routing\n","stream":"stderr","time":"2024-01-19T03:25:48.513548049Z"}
{"log":"2024-01-19T03:25:48Z ERR Request failed error=\"dial tcp 10.3.61.102:7: i/o timeout\" connIndex=3 dest=10.3.61.102:7 event=0 ip=198.41.192.167 type=tcp\n","stream":"stderr","time":"2024-01-19T03:25:48.513624882Z"}
{"log":"2024-01-19T03:25:58Z ERR  error=\"dial tcp 10.3.61.102:7: connect: no route to host\" event=2 flowID=069b8edd-985d-470b-af8f-ad762f778e52 originService=warp-routing\n","stream":"stderr","time":"2024-01-19T03:25:58.423148606Z"}
{"log":"2024-01-19T03:25:58Z ERR Request failed error=\"dial tcp 10.3.61.102:7: connect: no route to host\" connIndex=0 dest=10.3.61.102:7 event=0 ip=198.41.200.23 type=tcp\n","stream":"stderr","time":"2024-01-19T03:25:58.423201767Z"}
{"log":"2024-01-19T03:25:58Z ERR  error=\"dial tcp 10.3.61.101:7: connect: no route to host\" event=2 flowID=0de73855-09cf-44fc-82a4-8bb128d4ea9c originService=warp-routing\n","stream":"stderr","time":"2024-01-19T03:25:58.423209512Z"}
{"log":"2024-01-19T03:25:58Z ERR  error=\"dial tcp 10.3.61.102:7: connect: no route to host\" event=2 flowID=37711805-51d5-4035-9848-406af7fcf8be originService=warp-routing\n","stream":"stderr","time":"2024-01-19T03:25:58.423336627Z"}
{"log":"2024-01-19T03:25:58Z ERR Request failed error=\"dial tcp 10.3.61.102:7: connect: no route to host\" connIndex=0 dest=10.3.61.102:7 event=0 ip=198.41.200.23 type=tcp\n","stream":"stderr","time":"2024-01-19T03:25:58.423359573Z"}
{"log":"2024-01-19T03:25:58Z ERR Request failed error=\"dial tcp 10.3.61.101:7: connect: no route to host\" connIndex=0 dest=10.3.61.101:7 event=0 ip=198.41.200.23 type=tcp\n","stream":"stderr","time":"2024-01-19T03:25:58.423366962Z"}
{"log":"2024-01-19T03:29:01Z ERR  error=\"Incoming request ended abruptly: context canceled\" cfRay=847bfd732d41371d-FRA event=1 ingressRule=10 originService=!!!REDACTED!!!\n","stream":"stderr","time":"2024-01-19T03:29:01.776352194Z"}
{"log":"2024-01-19T03:29:01Z ERR Request failed error=\"Incoming request ended abruptly: context canceled\" connIndex=1 dest=!!!REDACTED!!! event=0 ip=198.41.192.227 type=http\n","stream":"stderr","time":"2024-01-19T03:29:01.776393877Z"}
{"log":"2024-01-19T03:29:21Z ERR  error=\"dial tcp 10.3.61.101:7: connect: no route to host\" event=2 flowID=4fe91433-8843-491e-9001-5c405ef3fd6f originService=warp-routing\n","stream":"stderr","time":"2024-01-19T03:29:21.543166379Z"}
{"log":"2024-01-19T03:29:21Z ERR Request failed error=\"dial tcp 10.3.61.101:7: connect: no route to host\" connIndex=2 dest=10.3.61.101:7 event=0 ip=198.41.200.233 type=tcp\n","stream":"stderr","time":"2024-01-19T03:29:21.543210954Z"}
{"log":"2024-01-19T03:29:21Z ERR  error=\"dial tcp 10.3.61.101:7: connect: no route to host\" event=2 flowID=0df54e71-59e3-4276-b931-eb312877c575 originService=warp-routing\n","stream":"stderr","time":"2024-01-19T03:29:21.569784453Z"}
{"log":"2024-01-19T03:29:21Z ERR Request failed error=\"dial tcp 10.3.61.101:7: connect: no route to host\" connIndex=2 dest=10.3.61.101:7 event=0 ip=198.41.200.233 type=tcp\n","stream":"stderr","time":"2024-01-19T03:29:21.56981571Z"}
{"log":"2024-01-19T03:29:21Z ERR  error=\"dial tcp 10.3.61.101:7: connect: no route to host\" event=2 flowID=62e6d700-1e95-4f89-a54f-9295e976dd71 originService=warp-routing\n","stream":"stderr","time":"2024-01-19T03:29:21.596872569Z"}
{"log":"2024-01-19T03:29:21Z ERR Request failed error=\"dial tcp 10.3.61.101:7: connect: no route to host\" connIndex=1 dest=10.3.61.101:7 event=0 ip=198.41.192.227 type=tcp\n","stream":"stderr","time":"2024-01-19T03:29:21.596980437Z"}
{"log":"2024-01-19T03:29:50Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.233\n","stream":"stderr","time":"2024-01-19T03:29:50.088967065Z"}
......
{"log":"2024-01-19T03:29:50Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.233\n","stream":"stderr","time":"2024-01-19T03:29:50.426202144Z"}
{"log":"2024-01-19T03:29:50Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.233\n","stream":"stderr","time":"2024-01-19T03:29:50.426207777Z"}
{"log":"2024-01-19T03:29:50Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.233\n","stream":"stderr","time":"2024-01-19T03:29:50.426652707Z"}
{"log":"2024-01-19T03:29:50Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.233\n","stream":"stderr","time":"2024-01-19T03:29:50.426840365Z"}
{"log":"2024-01-19T03:29:50Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.233\n","stream":"stderr","time":"2024-01-19T03:29:50.427000509Z"}
{"log":"2024-01-19T03:29:50Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.233\n","stream":"stderr","time":"2024-01-19T03:29:50.4270424Z"}
{"log":"2024-01-19T03:29:50Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.233\n","stream":"stderr","time":"2024-01-19T03:29:50.427134195Z"}
{"log":"2024-01-19T03:29:50Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.233\n","stream":"stderr","time":"2024-01-19T03:29:50.427282169Z"}
{"log":"2024-01-19T03:29:50Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.233\n","stream":"stderr","time":"2024-01-19T03:29:50.427427734Z"}
{"log":"2024-01-19T03:29:50Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.233\n","stream":"stderr","time":"2024-01-19T03:29:50.427757948Z"}
{"log":"2024-01-19T03:29:50Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.233\n","stream":"stderr","time":"2024-01-19T03:29:50.4278911Z"}
{"log":"2024-01-19T03:29:50Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.233\n","stream":"stderr","time":"2024-01-19T03:29:50.428563992Z"}
{"log":"2024-01-19T03:29:50Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.233\n","stream":"stderr","time":"2024-01-19T03:29:50.428577878Z"}
{"log":"2024-01-19T03:29:50Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.233\n","stream":"stderr","time":"2024-01-19T03:29:50.428654528Z"}
{"log":"2024-01-19T03:29:50Z WRN Failed to serve quic connection error=\"timeout: no recent network activity\" connIndex=2 event=0 ip=198.41.200.233\n","stream":"stderr","time":"2024-01-19T03:29:50.429230542Z"}
{"log":"2024-01-19T03:29:50Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.233\n","stream":"stderr","time":"2024-01-19T03:29:50.429857582Z"}
{"log":"2024-01-19T03:29:50Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.233\n","stream":"stderr","time":"2024-01-19T03:29:50.42990194Z"}
{"log":"2024-01-19T03:29:50Z WRN Serve tunnel error error=\"timeout: no recent network activity\" connIndex=2 event=0 ip=198.41.200.233\n","stream":"stderr","time":"2024-01-19T03:29:50.43048991Z"}
{"log":"2024-01-19T03:29:50Z INF Retrying connection in up to 1s connIndex=2 event=0 ip=198.41.200.233\n","stream":"stderr","time":"2024-01-19T03:29:50.430671463Z"}
{"log":"2024-01-19T03:29:50Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.233\n","stream":"stderr","time":"2024-01-19T03:29:50.430938746Z"}
{"log":"2024-01-19T03:29:50Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.233\n","stream":"stderr","time":"2024-01-19T03:29:50.431043266Z"}
{"log":"2024-01-19T03:29:50Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.233\n","stream":"stderr","time":"2024-01-19T03:29:50.431130879Z"}
{"log":"2024-01-19T03:29:50Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.233\n","stream":"stderr","time":"2024-01-19T03:29:50.431349191Z"}
......
{"log":"2024-01-19T03:29:50Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.233\n","stream":"stderr","time":"2024-01-19T03:29:51.242298907Z"}
{"log":"2024-01-19T03:29:50Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.233\n","stream":"stderr","time":"2024-01-19T03:29:51.257987468Z"}
{"log":"2024-01-19T03:29:50Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.233\n","stream":"stderr","time":"2024-01-19T03:29:51.386373981Z"}
{"log":"2024-01-19T03:29:50Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.233\n","stream":"stderr","time":"2024-01-19T03:29:51.442223954Z"}
{"log":"2024-01-19T03:29:51Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.233\n","stream":"stderr","time":"2024-01-19T03:29:51.442531718Z"}
{"log":"2024-01-19T03:29:52Z WRN Connection terminated error=\"timeout: no recent network activity\" connIndex=2\n","stream":"stderr","time":"2024-01-19T03:29:53.135684172Z"}
{"log":"2024-01-19T03:30:07Z INF Registered tunnel connection connIndex=2 connection=5a92af8f-66f1-420c-a3d3-09cb7e3f8077 event=0 ip=198.41.200.73 location=vie02 protocol=quic\n","stream":"stderr","time":"2024-01-19T03:30:07.891099018Z"}
{"log":"2024-01-19T03:31:53Z ERR  error=\"dial tcp 192.168.1.145:7000: i/o timeout\" event=2 flowID=d28e6e60-50c1-4ce5-bae2-711353d06246 originService=warp-routing\n","stream":"stderr","time":"2024-01-19T03:31:54.112531337Z"}
{"log":"2024-01-19T03:31:54Z ERR Request failed error=\"dial tcp 192.168.1.145:7000: i/o timeout\" connIndex=0 dest=192.168.1.145:7000 event=0 ip=198.41.200.23 type=tcp\n","stream":"stderr","time":"2024-01-19T03:31:54.112576587Z"}
{"log":"2024-01-19T03:37:01Z ERR  error=\"Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: net/http: TLS handshake timeout\" cfRay=847c09121e535b0f-VIE event=1 ingressRule=7 originService=!!!REDACTED!!!\n","stream":"stderr","time":"2024-01-19T03:37:01.930280366Z"}
{"log":"2024-01-19T03:37:01Z ERR Request failed error=\"Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: net/http: TLS handshake timeout\" connIndex=0 dest=!!!REDACTED!!! event=0 ip=198.41.200.23 type=http\n","stream":"stderr","time":"2024-01-19T03:37:01.930377796Z"}
{"log":"2024-01-19T03:38:32Z ERR  error=\"Incoming request ended abruptly: context canceled\" cfRay=847c0b3ccc0379c7-LHR event=1 ingressRule=10 originService=!!!REDACTED!!!\n","stream":"stderr","time":"2024-01-19T03:38:32.355257936Z"}
{"log":"2024-01-19T03:38:32Z ERR Request failed error=\"Incoming request ended abruptly: context canceled\" connIndex=1 dest=!!!REDACTED!!! event=0 ip=198.41.192.227 type=http\n","stream":"stderr","time":"2024-01-19T03:38:32.360531132Z"}

Additional context
Omitted many lines of the Datagram session unregistration in-between.
Also something I noted: version 2024.1.2 always only connects to the VIE colo, whereas 2024.1.3 somehow always chooses BUD as well.
The no recent network activity error seems very suspicious to me.

Really tried to find any other errors on my side, but I can't find anything.

João "Pisco" Fernandes · Answer 1 · Sat Jan 20 2024 00:03:17 GMT+0800 (China Standard Time)

Hi @nwpr,
We see 2 cloudflared instances running your tunnel, one with the most recent version 2024.1.3 that has 4 connections, and another running 2024.1.2 with only a single connection. Are they running in the same environment? Did you manually forced the old client to only perform 1 connection?

Nikolaus Walther · Answer 2 · Sat Jan 20 2024 00:13:29 GMT+0800 (China Standard Time)

Hi @jcsf,
Yes, the instances of this tunnel are running in the same environment and they do have the exact same docker-compose file - except the version tag and ip address.

Also the other tunnel (e1a33431-1889-483f-9677-235a944fbc07) is running in a different country, running on ARM64, directly on Ubuntu, without Docker, and the error logs look quite similar. This one is without replica, and the connection here is also flapping since the version update.

João "Pisco" Fernandes · Answer 3 · Sat Jan 20 2024 00:40:08 GMT+0800 (China Standard Time)

The instances running on docker are running on the same host or different hosts? Or how are you setting the different ip addresses, because we would expect 4 connections to come from the older version and it is only doing 1 connection.

Also, which docker image are you using? The official one?

Finally, could you perform a test for us. In your docker compose file with the latest version add the following environment variable to the docker container and restart it.

QUIC_GO_DISABLE_GSO=1

version: "3.4"

services:

  cloudflared:
    ...
    environment:
      - QUIC_GO_DISABLE_GSO=1
    ...

Nikolaus Walther · Answer 4 · Sat Jan 20 2024 03:37:44 GMT+0800 (China Standard Time)

The instances running on docker are running on the same host or different hosts? Or how are you setting the different ip addresses, because we would expect 4 connections to come from the older version and it is only doing 1 connection.

They are running on the same host. The docker file now looks a follows for the latest version:

version: "3.7"

services:
  cloudflared:
    image: cloudflare/cloudflared
    container_name: cloudflared
    command: "tunnel --no-autoupdate run --token !!!REDACTED!!!"
    restart: always
    environment:
      - QUIC_GO_DISABLE_GSO=1
    networks:
      server:
        ipv4_address: 10.10.11.22
networks:
  server:
    external: true
    name: server

And like this on the 2024.1.2:

version: "3.7"

services:
  cloudflared:
    image: cloudflare/cloudflared:2024.1.2
    container_name: cloudflared-repl
    command: "tunnel --no-autoupdate run --token !!!REDACTED!!!"
    restart: always
    networks:
      server:
        ipv4_address: 10.10.12.22
networks:
  server:
    external: true
    name: server

I think the v2024.1.2 should now have all connections, after restarting the container. Don't know what went wrong there, but the restart seems to have solved this specific problem.

Finally, could you perform a test for us. In your docker compose file with the latest version add the following environment variable to the docker container and restart it.

QUIC_GO_DISABLE_GSO=1

I've now added the flag, as seen above, but the connection issue and flapping still exists.

2024-01-19T19:19:09Z ERR Datagram session unregistration timeout connIndex=3 event=0 ip=198.41.192.7
2024-01-19T19:19:09Z INF Unregistered tunnel connection connIndex=3 event=0 ip=198.41.192.7
2024-01-19T19:19:09Z WRN Failed to serve quic connection error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=3 event=0 ip=198.41.192.7
2024-01-19T19:19:09Z WRN Serve tunnel error error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=3 event=0 ip=198.41.192.7
2024-01-19T19:19:09Z INF Retrying connection in up to 1s connIndex=3 event=0 ip=198.41.192.7
2024-01-19T19:19:09Z WRN Connection terminated error="timeout: no recent network activity" connIndex=1
2024-01-19T19:19:10Z WRN Connection terminated error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=3
2024-01-19T19:19:30Z INF Unregistered tunnel connection connIndex=2 event=0 ip=198.41.200.63
2024-01-19T19:19:30Z WRN Failed to serve quic connection error="context canceled" connIndex=2 event=0 ip=198.41.200.63
2024-01-19T19:19:30Z INF Retrying connection in up to 1s connIndex=2 event=0 ip=198.41.200.63
2024-01-19T19:19:35Z WRN Connection terminated error="context canceled" connIndex=2
2024-01-19T19:20:15Z ERR Datagram session unregistration timeout connIndex=0 event=0 ip=198.41.200.13
2024-01-19T19:20:15Z INF Unregistered tunnel connection connIndex=0 event=0 ip=198.41.200.13
2024-01-19T19:20:15Z ERR Failed to serve quic connection error="context canceled" connIndex=0 event=0 ip=198.41.200.13
2024-01-19T19:20:17Z ERR  error="Error writing response header: timeout: no recent network activity" cfRay=84816e937ed8b051-ATL event=1 ingressRule=10 originService=!!!REDACTED!!!
2024-01-19T19:20:17Z ERR Request failed error="Error writing response header: timeout: no recent network activity" connIndex=0 dest=!!!REDACTED!!! event=0 ip=198.41.200.13 type=http
2024-01-19T19:20:17Z INF Retrying connection in up to 1s connIndex=0 event=0 ip=198.41.200.13
2024-01-19T19:20:21Z ERR Connection terminated error="context canceled" connIndex=0
2024-01-19T19:20:31Z INF Registered tunnel connection connIndex=3 connection=51e003c6-dd94-4693-87a1-b68c44aa2c8a event=0 ip=198.41.192.7 location=bud01 protocol=quic
2024-01-19T19:20:35Z INF Registered tunnel connection connIndex=2 connection=036bc025-afbf-4ef5-be39-ea8d0d549985 event=0 ip=198.41.200.63 location=vie02 protocol=quic
2024-01-19T19:20:41Z INF Registered tunnel connection connIndex=1 connection=39bead21-7ec7-4d58-83fa-617d2791b479 event=0 ip=198.41.192.107 location=bud01 protocol=quic
2024-01-19T19:20:41Z INF Unregistered tunnel connection connIndex=2 event=0 ip=198.41.200.63
2024-01-19T19:20:41Z ERR writing release: Application error 0x0 (remote)
2024-01-19T19:20:41Z WRN Failed to serve quic connection error="failed to accept QUIC stream: Application error 0x0 (remote)" connIndex=2 event=0 ip=198.41.200.63
2024-01-19T19:20:41Z WRN Serve tunnel error error="failed to accept QUIC stream: Application error 0x0 (remote)" connIndex=2 event=0 ip=198.41.200.63
2024-01-19T19:20:41Z INF Retrying connection in up to 1s connIndex=2 event=0 ip=198.41.200.63
2024-01-19T19:20:42Z WRN Connection terminated error="failed to accept QUIC stream: Application error 0x0 (remote)" connIndex=2
2024-01-19T19:20:48Z INF Unregistered tunnel connection connIndex=1 event=0 ip=198.41.192.107
2024-01-19T19:20:49Z WRN Failed to serve quic connection error="timeout: no recent network activity" connIndex=1 event=0 ip=198.41.192.107
2024-01-19T19:20:51Z WRN Serve tunnel error error="timeout: no recent network activity" connIndex=1 event=0 ip=198.41.192.107
2024-01-19T19:20:51Z INF Retrying connection in up to 1s connIndex=1 event=0 ip=198.41.192.107
2024-01-19T19:20:53Z WRN Connection terminated error="timeout: no recent network activity" connIndex=1
2024-01-19T19:20:55Z INF Registered tunnel connection connIndex=0 connection=477c3669-b1e4-4080-a00e-e9bec26a019c event=0 ip=198.41.200.13 location=vie06 protocol=quic
2024-01-19T19:21:00Z INF Registered tunnel connection connIndex=2 connection=27bc95a8-f20c-49ca-bc24-e2ac7a69642e event=0 ip=198.41.200.63 location=vie02 protocol=quic
2024-01-19T19:21:01Z INF Unregistered tunnel connection connIndex=0 event=0 ip=198.41.200.13
2024-01-19T19:21:01Z WRN Failed to serve quic connection error="Application error 0x0 (remote)" connIndex=0 event=0 ip=198.41.200.13
2024-01-19T19:21:01Z WRN Serve tunnel error error="Application error 0x0 (remote)" connIndex=0 event=0 ip=198.41.200.13
2024-01-19T19:21:01Z INF Retrying connection in up to 1s connIndex=0 event=0 ip=198.41.200.13
2024-01-19T19:21:02Z WRN Connection terminated error="Application error 0x0 (remote)" connIndex=0
2024-01-19T19:21:39Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.63
2024-01-19T19:21:39Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.63
2024-01-19T19:21:39Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.63
2024-01-19T19:21:39Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.63
2024-01-19T19:21:39Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.63
2024-01-19T19:21:39Z INF Unregistered tunnel connection connIndex=2 event=0 ip=198.41.200.63
2024-01-19T19:21:39Z WRN Failed to serve quic connection error="timeout: no recent network activity" connIndex=2 event=0 ip=198.41.200.63
2024-01-19T19:21:39Z WRN Serve tunnel error error="timeout: no recent network activity" connIndex=2 event=0 ip=198.41.200.63
2024-01-19T19:21:39Z INF Retrying connection in up to 1s connIndex=2 event=0 ip=198.41.200.63
2024-01-19T19:21:39Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.63
2024-01-19T19:21:40Z WRN Connection terminated error="timeout: no recent network activity" connIndex=2
2024-01-19T19:22:03Z ERR Datagram session unregistration timeout connIndex=3 event=0 ip=198.41.192.7
2024-01-19T19:22:03Z ERR Datagram session unregistration timeout connIndex=3 event=0 ip=198.41.192.7
2024-01-19T19:22:03Z INF Unregistered tunnel connection connIndex=3 event=0 ip=198.41.192.7
2024-01-19T19:22:03Z ERR Failed to serve quic connection error="timeout: no recent network activity" connIndex=3 event=0 ip=198.41.192.7
2024-01-19T19:22:06Z ERR Serve tunnel error error="timeout: no recent network activity" connIndex=3 event=0 ip=198.41.192.7
2024-01-19T19:22:06Z INF Retrying connection in up to 1s connIndex=3 event=0 ip=198.41.192.7
2024-01-19T19:22:09Z ERR Connection terminated error="timeout: no recent network activity" connIndex=3
2024-01-19T19:22:26Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: handshake did not complete in time" connIndex=2 event=0 ip=198.41.200.53
2024-01-19T19:22:26Z INF Retrying connection in up to 4s connIndex=2 event=0 ip=198.41.200.53
2024-01-19T19:22:26Z INF Registered tunnel connection connIndex=0 connection=9fae9e99-cb13-421b-8eef-19cb817fe25d event=0 ip=198.41.200.13 location=vie05 protocol=quic
2024-01-19T19:22:29Z WRN Connection terminated error="failed to dial to edge with quic: timeout: handshake did not complete in time" connIndex=2
2024-01-19T19:22:37Z INF Registered tunnel connection connIndex=1 connection=a0822f22-c0ad-4c80-88ca-e02b3b3a31c2 event=0 ip=198.41.192.27 location=bud01 protocol=quic
2024-01-19T19:23:28Z INF Unregistered tunnel connection connIndex=0 event=0 ip=198.41.200.13
2024-01-19T19:23:29Z WRN Failed to serve quic connection error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.13
2024-01-19T19:23:29Z WRN Serve tunnel error error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.13
2024-01-19T19:23:29Z INF Retrying connection in up to 1s connIndex=0 event=0 ip=198.41.200.13
2024-01-19T19:23:30Z WRN Connection terminated error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=0
2024-01-19T19:23:32Z ERR  error="timeout: no recent network activity" event=2 flowID=67d91d4c-661f-4134-bdcf-2d586eb2bb3d originService=warp-routing
2024-01-19T19:23:33Z INF Unregistered tunnel connection connIndex=1 event=0 ip=198.41.192.27
2024-01-19T19:23:33Z ERR Failed to serve quic connection error="timeout: no recent network activity" connIndex=1 event=0 ip=198.41.192.27
2024-01-19T19:23:33Z ERR Serve tunnel error error="timeout: no recent network activity" connIndex=1 event=0 ip=198.41.192.27
2024-01-19T19:23:34Z INF Retrying connection in up to 1s connIndex=1 event=0 ip=198.41.192.27
2024-01-19T19:23:32Z ERR Request failed error="timeout: no recent network activity" connIndex=1 dest=10.3.61.102:7 event=0 ip=198.41.192.27 type=tcp
2024-01-19T19:23:36Z ERR Connection terminated error="timeout: no recent network activity" connIndex=1
2024-01-19T19:24:10Z INF Registered tunnel connection connIndex=1 connection=0e052dbb-bc1c-43bb-b571-9b6d3d12ea1f event=0 ip=198.41.192.167 location=bud01 protocol=quic
2024-01-19T19:24:14Z INF Registered tunnel connection connIndex=0 connection=1233b3ab-6c0c-40ac-9a50-115516a60c01 event=0 ip=198.41.200.13 location=vie05 protocol=quic
2024-01-19T19:24:16Z INF Registered tunnel connection connIndex=3 connection=44685b59-865f-4097-9d72-95b16f521ddb event=0 ip=198.41.192.107 location=bud01 protocol=quic
2024-01-19T19:24:23Z INF Registered tunnel connection connIndex=2 connection=93b6d00a-ef70-4f47-ace9-eb1093c57e49 event=0 ip=198.41.200.233 location=vie06 protocol=quic
2024-01-19T19:26:06Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.233
2024-01-19T19:26:06Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.233
2024-01-19T19:26:06Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.233
2024-01-19T19:26:07Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.233
2024-01-19T19:26:07Z INF Unregistered tunnel connection connIndex=2 event=0 ip=198.41.200.233
2024-01-19T19:26:07Z WRN Failed to serve quic connection error="timeout: no recent network activity" connIndex=2 event=0 ip=198.41.200.233
2024-01-19T19:26:08Z WRN Serve tunnel error error="timeout: no recent network activity" connIndex=2 event=0 ip=198.41.200.233
2024-01-19T19:26:09Z INF Retrying connection in up to 1s connIndex=2 event=0 ip=198.41.200.233
2024-01-19T19:26:11Z WRN Connection terminated error="timeout: no recent network activity" connIndex=2
2024-01-19T19:26:53Z INF Unregistered tunnel connection connIndex=0 event=0 ip=198.41.200.13
2024-01-19T19:26:54Z ERR Datagram session unregistration timeout connIndex=0 event=0 ip=198.41.200.13
2024-01-19T19:26:54Z WRN Failed to serve quic connection error="timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.13
2024-01-19T19:26:56Z WRN Serve tunnel error error="timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.13
2024-01-19T19:26:56Z INF Retrying connection in up to 1s connIndex=0 event=0 ip=198.41.200.13
2024-01-19T19:26:58Z WRN Connection terminated error="timeout: no recent network activity" connIndex=0
2024-01-19T19:27:48Z ERR  error="Incoming request ended abruptly: context canceled" cfRay=84817a04bbc75fc8-SIN event=1 ingressRule=10 originService=!!!REDACTED!!!
2024-01-19T19:27:48Z ERR Request failed error="Incoming request ended abruptly: context canceled" connIndex=1 dest=!!!REDACTED!!! event=0 ip=198.41.192.167 type=http
2024-01-19T19:27:49Z ERR  error="Incoming request ended abruptly: context canceled" cfRay=848179e1981bc60d-KHI event=1 ingressRule=10 originService=!!!REDACTED!!!
2024-01-19T19:27:49Z ERR Request failed error="Incoming request ended abruptly: context canceled" connIndex=3 dest=!!!REDACTED!!! event=0 ip=198.41.192.107 type=http
2024-01-19T19:28:21Z INF Registered tunnel connection connIndex=0 connection=cedf5778-2e65-4075-ae74-ae7496cf3f66 event=0 ip=198.41.200.53 location=vie06 protocol=quic
2024-01-19T19:28:44Z INF Registered tunnel connection connIndex=2 connection=b82ab22d-a42e-4002-b45a-9371cd598eb1 event=0 ip=198.41.200.33 location=vie05 protocol=quic
2024-01-19T19:31:14Z ERR Datagram session unregistration timeout connIndex=2 event=0 ip=198.41.200.33
2024-01-19T19:31:14Z INF Unregistered tunnel connection connIndex=2 event=0 ip=198.41.200.33
2024-01-19T19:31:14Z WRN Failed to serve quic connection error="timeout: no recent network activity" connIndex=2 event=0 ip=198.41.200.33
2024-01-19T19:31:15Z WRN Serve tunnel error error="timeout: no recent network activity" connIndex=2 event=0 ip=198.41.200.33
2024-01-19T19:31:17Z INF Retrying connection in up to 1s connIndex=2 event=0 ip=198.41.200.33
2024-01-19T19:31:18Z WRN Connection terminated error="timeout: no recent network activity" connIndex=2

João "Pisco" Fernandes · Answer 5 · Sat Jan 20 2024 04:05:55 GMT+0800 (China Standard Time)

We had more clients complaining with issues on cloudflared and actually found the issue. We are releasing the fix in the next hours. Your tunnel is proxying private traffic so I believe it might be affecting you. Please test the new version once it's up and tell us if it is fixed.

João "Pisco" Fernandes · Answer 6 · Sat Jan 20 2024 04:36:08 GMT+0800 (China Standard Time)

Version Released: 2024.1.4

Nikolaus Walther · Answer 7 · Sat Jan 20 2024 04:58:26 GMT+0800 (China Standard Time)

Connection seems to be working great now!
Nevertheless, I'll let it run for a few hours before closing this issue, just to be sure.

Thanks for the fast fix 👍

Nikolaus Walther · Answer 8 · Sat Jan 20 2024 06:30:06 GMT+0800 (China Standard Time)

Can confirm it's working and the issue is resolved. Thank you!