🐛 BUG: Unable to ping the lighthouse, all connections are disconnected.
QZAiXH opened this issue · comments
What version of nebula are you using?
Release v1.7.3-nightly20231018
What operating system are you using?
Linux, Windows
Describe the Bug
I am using a Windows computer (with IP address 172.17.3.20) to connect to Linux (with IP address 172.17.2.121). However, during the usage, I suddenly encountered a disconnection issue. I checked the logs of Linux and found an error message saying ‘Failed to write outgoing packet’, and there were no other outputs in the log afterwards. At this point, the Linux machine cannot ping the lighthouse, and all connections are disconnected. I suspect this may be related to #1007. Since Windows is running in the background and does not log, I cannot provide more information.
Logs from affected hosts
linux log
time="2023-11-17T10:02:18+08:00" level=info msg="Attempt to relay through hosts" localIndex=768454828 relays="[172.17.0.1]" remoteIndex=0 vpnIp=172.17.3.20
time="2023-11-17T10:02:18+08:00" level=info msg="Send handshake via relay" localIndex=768454828 relay=172.17.0.1 remoteIndex=0 vpnIp=172.17.3.20
time="2023-11-17T10:02:18+08:00" level=info msg="Handshake message received" certName=J1exwzkmO3 fingerprint=13ee1fc8dcbeef6b9736ab410e6cdd99d51f9ce5b7fd5fb8d4474b630c920c95 handshake="map[stage:1 style:ix_psk0]" initiatorIndex=1659627004 issuer=c670f7dcc09a02343cfab50c164fd8ebd201ab21852d5202f881c604fd2e5680 remoteIndex=0 responderIndex=0 udpAddr="183.236.13.195:2060" vpnIp=172.17.3.20
time="2023-11-17T10:02:18+08:00" level=info msg="Handshake message sent" certName=J1exwzkmO3 fingerprint=13ee1fc8dcbeef6b9736ab410e6cdd99d51f9ce5b7fd5fb8d4474b630c920c95 handshake="map[stage:2 style:ix_psk0]" initiatorIndex=1659627004 issuer=c670f7dcc09a02343cfab50c164fd8ebd201ab21852d5202f881c604fd2e5680 remoteIndex=0 responderIndex=1639135172 sentCachedPackets=0 udpAddr="183.236.13.195:2060" vpnIp=172.17.3.20
time="2023-11-17T10:02:18+08:00" level=info msg="Handshake message received" certName=J1exwzkmO3 fingerprint=13ee1fc8dcbeef6b9736ab410e6cdd99d51f9ce5b7fd5fb8d4474b630c920c95 handshake="map[stage:1 style:ix_psk0]" initiatorIndex=1659627004 issuer=c670f7dcc09a02343cfab50c164fd8ebd201ab21852d5202f881c604fd2e5680 remoteIndex=0 responderIndex=0 udpAddr="<nil>" vpnIp=172.17.3.20
time="2023-11-17T10:02:18+08:00" level=info msg="Handshake message sent" cached=true handshake="map[stage:2 style:ix_psk0]" relay=172.17.0.1 vpnIp=172.17.3.20
time="2023-11-17T10:02:19+08:00" level=info msg="Handshake message received" certName=J1exwzkmO3 durationNs=44635159 fingerprint=13ee1fc8dcbeef6b9736ab410e6cdd99d51f9ce5b7fd5fb8d4474b630c920c95 handshake="map[stage:2 style:ix_psk0]" initiatorIndex=768454828 issuer=c670f7dcc09a02343cfab50c164fd8ebd201ab21852d5202f881c604fd2e5680 remoteIndex=768454828 responderIndex=2967210750 sentCachedPackets=1 udpAddr="<nil>" vpnIp=172.17.3.20
time="2023-11-17T10:02:27+08:00" level=error msg="Failed to write outgoing packet" certName=J1exwzkmO3 error="sendto: network is unreachable" localIndex=768454828 remoteIndex=2967210750 udpAddr="[fd00:485f:824:537::1011]:62584" vpnIp=172.17.3.20
time="2023-11-17T10:02:27+08:00" level=info msg="Host roamed to new udp ip/port." certName=J1exwzkmO3 localIndex=768454828 newAddr="183.236.13.195:2060" remoteIndex=2967210750 udpAddr="<nil>" vpnIp=172.17.3.20
time="2023-11-17T10:03:12+08:00" level=info msg="Handshake timed out" durationNs=6498417601 handshake="map[stage:1 style:ix_psk0]" initiatorIndex=1609970695 localIndex=1609970695 remoteIndex=0 udpAddrs="[]" vpnIp=172.17.3.20
lighthouse log
Nov 17 09:41:48 zh01 nebula[729]: time="2023-11-17T09:41:48+08:00" level=info msg=handleCreateRelayRequest initiatorRelayIndex=900985918 relayFrom=172.17.3.18 relayTo=172.17.2.121 vpnIp=172.17.3.18
Nov 17 09:41:48 zh01 nebula[729]: time="2023-11-17T09:41:48+08:00" level=info msg="send CreateRelayRequest" initiatorRelayIndex=4017635712 relayFrom=172.17.3.18 relayTo=172.17.2.121 responderRelayIndex=0 vpnIp=172.17.2.121
Nov 17 09:41:48 zh01 nebula[729]: time="2023-11-17T09:41:48+08:00" level=info msg=handleCreateRelayResponse initiatorRelayIndex=4017635712 relayFrom=172.17.3.18 relayTo=172.17.2.121 responderRelayIndex=2669366842 vpnIp=172.17.2.121
Nov 17 09:41:48 zh01 nebula[729]: time="2023-11-17T09:41:48+08:00" level=info msg="send CreateRelayResponse" initiatorRelayIndex=900985918 relayFrom=172.17.3.18 relayTo=172.17.2.121 responderRelayIndex=1408905089 vpnIp=172.17.3.18
Nov 17 09:44:24 zh01 nebula[729]: time="2023-11-17T09:44:24+08:00" level=info msg="Failed to find target host info by ip" certName=gNANVz7gEh error="unable to find host" localIndex=2172632246 relayTo=172.17.2.121 remoteIndex=3114150245 vpnIp=172.17.3.18
Nov 17 09:44:25 zh01 nebula[729]: time="2023-11-17T09:44:25+08:00" level=info msg="Failed to find target host info by ip" certName=gNANVz7gEh error="unable to find host" localIndex=2172632246 relayTo=172.17.2.121 remoteIndex=3114150245 vpnIp=172.17.3.18
Nov 17 09:44:27 zh01 nebula[729]: time="2023-11-17T09:44:27+08:00" level=info msg="Failed to find target host info by ip" certName=gNANVz7gEh error="unable to find host" localIndex=2172632246 relayTo=172.17.2.121 remoteIndex=3114150245 vpnIp=172.17.3.18
Nov 17 09:44:32 zh01 nebula[729]: time="2023-11-17T09:44:32+08:00" level=info msg="Failed to find target host info by ip" certName=gNANVz7gEh error="unable to find host" localIndex=2172632246 relayTo=172.17.2.121 remoteIndex=3114150245 vpnIp=172.17.3.18
Nov 17 09:44:34 zh01 nebula[729]: time="2023-11-17T09:44:34+08:00" level=info msg="Failed to find target host info by ip" certName=gNANVz7gEh error="unable to find host" localIndex=2172632246 relayTo=172.17.2.121 remoteIndex=3114150245 vpnIp=172.17.3.18
Nov 17 09:44:45 zh01 nebula[729]: time="2023-11-17T09:44:45+08:00" level=info msg="Failed to find target host info by ip" certName=J1exwzkmO3 error="unable to find host" localIndex=3857821913 relayTo=172.17.2.121 remoteIndex=3064253789 vpnIp=172.17.3.20
Nov 17 09:44:45 zh01 nebula[729]: time="2023-11-17T09:44:45+08:00" level=info msg="Failed to find target host info by ip" certName=J1exwzkmO3 error="unable to find host" localIndex=3857821913 relayTo=172.17.2.121 remoteIndex=3064253789 vpnIp=172.17.3.20
Nov 17 09:44:46 zh01 nebula[729]: time="2023-11-17T09:44:46+08:00" level=info msg="Failed to find target host info by ip" certName=J1exwzkmO3 error="unable to find host" localIndex=3857821913 relayTo=172.17.2.121 remoteIndex=3064253789 vpnIp=172.17.3.20
Nov 17 09:44:47 zh01 nebula[729]: time="2023-11-17T09:44:47+08:00" level=info msg="Failed to find target host info by ip" certName=J1exwzkmO3 error="unable to find host" localIndex=3857821913 relayTo=172.17.2.121 remoteIndex=3064253789 vpnIp=172.17.3.20
Nov 17 09:44:48 zh01 nebula[729]: time="2023-11-17T09:44:48+08:00" level=info msg="Failed to find target host info by ip" certName=J1exwzkmO3 error="unable to find host" localIndex=3857821913 relayTo=172.17.2.121 remoteIndex=3064253789 vpnIp=172.17.3.20
Nov 17 09:44:49 zh01 nebula[729]: time="2023-11-17T09:44:49+08:00" level=info msg="Failed to find target host info by ip" certName=J1exwzkmO3 error="unable to find host" localIndex=3857821913 relayTo=172.17.2.121 remoteIndex=3064253789 vpnIp=172.17.3.20
Nov 17 09:44:50 zh01 nebula[729]: time="2023-11-17T09:44:50+08:00" level=info msg="Failed to find target host info by ip" certName=J1exwzkmO3 error="unable to find host" localIndex=3857821913 relayTo=172.17.2.121 remoteIndex=3064253789 vpnIp=172.17.3.20
Nov 17 09:44:51 zh01 nebula[729]: time="2023-11-17T09:44:51+08:00" level=info msg="Failed to find target host info by ip" certName=J1exwzkmO3 error="unable to find host" localIndex=3857821913 relayTo=172.17.2.121 remoteIndex=3064253789 vpnIp=172.17.3.20
Nov 17 09:44:52 zh01 nebula[729]: time="2023-11-17T09:44:52+08:00" level=info msg="Failed to find target host info by ip" certName=J1exwzkmO3 error="unable to find host" localIndex=3857821913 relayTo=172.17.2.121 remoteIndex=3064253789 vpnIp=172.17.3.20
Nov 17 09:44:53 zh01 nebula[729]: time="2023-11-17T09:44:53+08:00" level=info msg="Failed to find target host info by ip" certName=J1exwzkmO3 error="unable to find host" localIndex=3857821913 relayTo=172.17.2.121 remoteIndex=3064253789 vpnIp=172.17.3.20
Nov 17 09:44:54 zh01 nebula[729]: time="2023-11-17T09:44:54+08:00" level=info msg="Failed to find target host info by ip" certName=J1exwzkmO3 error="unable to find host" localIndex=3857821913 relayTo=172.17.2.121 remoteIndex=3064253789 vpnIp=172.17.3.20
Nov 17 09:44:55 zh01 nebula[729]: time="2023-11-17T09:44:55+08:00" level=info msg="Failed to find target host info by ip" certName=J1exwzkmO3 error="unable to find host" localIndex=3857821913 relayTo=172.17.2.121 remoteIndex=3064253789 vpnIp=172.17.3.20
Nov 17 09:44:55 zh01 nebula[729]: time="2023-11-17T09:44:55+08:00" level=info msg="Failed to find target host info by ip" certName=J1exwzkmO3 error="unable to find host" localIndex=3857821913 relayTo=172.17.2.121 remoteIndex=3064253789 vpnIp=172.17.3.20
Nov 17 09:44:56 zh01 nebula[729]: time="2023-11-17T09:44:56+08:00" level=info msg="Failed to find target host info by ip" certName=J1exwzkmO3 error="unable to find host" localIndex=3857821913 relayTo=172.17.2.121 remoteIndex=3064253789 vpnIp=172.17.3.20
Nov 17 09:45:17 zh01 nebula[729]: time="2023-11-17T09:45:17+08:00" level=info msg="Handshake message received" certName=22eDichMk5 fingerprint=20e9dee44564e6c3494a2285628b551210b645d55785f32353bf2d41269ab395 handshake="map[stage:1 style:ix_psk0]" initiatorIndex=1691456506 issuer=c670f7dcc09a02343cfab50c164fd8ebd201ab21852d5202f881c604fd2e5680 remoteIndex=0 responderIndex=0 udpAddr="163.204.234.223:6151" vpnIp=172.17.2.121
Nov 17 09:45:17 zh01 nebula[729]: time="2023-11-17T09:45:17+08:00" level=info msg="Handshake message sent" certName=22eDichMk5 fingerprint=20e9dee44564e6c3494a2285628b551210b645d55785f32353bf2d41269ab395 handshake="map[stage:2 style:ix_psk0]" initiatorIndex=1691456506 issuer=c670f7dcc09a02343cfab50c164fd8ebd201ab21852d5202f881c604fd2e5680 remoteIndex=0 responderIndex=2329967232 sentCachedPackets=0 udpAddr="163.204.234.223:6151" vpnIp=172.17.2.121
Nov 17 09:45:18 zh01 nebula[729]: time="2023-11-17T09:45:18+08:00" level=info msg="Failed to find target host info by ip" certName=J1exwzkmO3 error="unable to find host with relay" localIndex=3857821913 relayTo=172.17.2.121 remoteIndex=3064253789 vpnIp=172.17.3.20
Nov 17 09:45:18 zh01 nebula[729]: time="2023-11-17T09:45:18+08:00" level=info msg="Failed to find target host info by ip" certName=gNANVz7gEh error="unable to find host with relay" localIndex=2172632246 relayTo=172.17.2.121 remoteIndex=3114150245 vpnIp=172.17.3.18
Nov 17 09:45:48 zh01 nebula[729]: time="2023-11-17T09:45:48+08:00" level=info msg="Failed to find target host info by ip" certName=gNANVz7gEh error="unable to find host with relay" localIndex=2172632246 relayTo=172.17.2.121 remoteIndex=3114150245 vpnIp=172.17.3.18
Nov 17 09:47:16 zh01 nebula[729]: time="2023-11-17T09:47:16+08:00" level=info msg=handleCreateRelayRequest initiatorRelayIndex=4180006229 relayFrom=172.17.2.121 relayTo=172.17.3.18 vpnIp=172.17.2.121
Nov 17 09:47:16 zh01 nebula[729]: time="2023-11-17T09:47:16+08:00" level=info msg=handleCreateRelayRequest initiatorRelayIndex=4180006229 relayFrom=172.17.2.121 relayTo=172.17.3.18 vpnIp=172.17.2.121
Nov 17 09:47:16 zh01 nebula[729]: time="2023-11-17T09:47:16+08:00" level=info msg="send CreateRelayResponse" initiatorRelayIndex=4180006229 relayFrom=172.17.2.121 relayTo=172.17.3.18 responderRelayIndex=2318876637 vpnIp=172.17.2.121
Nov 17 09:55:03 zh01 nebula[729]: time="2023-11-17T09:55:03+08:00" level=info msg=handleCreateRelayRequest initiatorRelayIndex=481910009 relayFrom=172.17.2.121 relayTo=172.17.3.20 vpnIp=172.17.2.121
Nov 17 09:55:03 zh01 nebula[729]: time="2023-11-17T09:55:03+08:00" level=info msg=handleCreateRelayRequest initiatorRelayIndex=481910009 relayFrom=172.17.2.121 relayTo=172.17.3.20 vpnIp=172.17.2.121
Nov 17 09:55:03 zh01 nebula[729]: time="2023-11-17T09:55:03+08:00" level=info msg="send CreateRelayResponse" initiatorRelayIndex=481910009 relayFrom=172.17.2.121 relayTo=172.17.3.20 responderRelayIndex=1288908194 vpnIp=172.17.2.121
Config files from affected hosts
No response
Sorry, I found that the version of lighthouse is different from linux and windows
Hi @QZAiXH -
Just FYI, the following log line is expected on IPv4-only hosts when they try to communicate with an IPv6-enabled host:
time="2023-11-17T10:02:27+08:00" level=error msg="Failed to write outgoing packet" certName=J1exwzkmO3 error="sendto: network is unreachable" localIndex=768454828 remoteIndex=2967210750 udpAddr="[fd00:485f:824:537::1011]:62584" vpnIp=172.17.3.20
Would you mind clarifying your solution here?
Sorry, I found that the version of lighthouse is different from linux and windows
Are you stating that it was running a different Nebula version, or the config was incorrect? As much detail as you can provide would be helpful.
Thanks!