🐛 BUG: Client to Client connection doesn't work

[peter0322]

What version of nebula are you using?

1.8.2

What operating system are you using?

Linux, Windows

Describe the Bug

We have configured a new network using nebula recently.

We have two servers as lighthouses. Other devices and pcs are nodes.

The certificates are created with below command.
./nebula-cert sign -name "lighthouse-1" -ip "10.89.12.1/16"
./nebula-cert sign -name "lighthouse-2" -ip "10.89.12.2/16"

./nebula-cert sign -name "client-1" -ip "10.89.15.1/16"
./nebula-cert sign -name "client-2" -ip "10.89.15.2/16"

Now we don't have issues for connection between the lighthouses and the nodes.
So for example, if I ping from the client 1 to lighthouse 1, it's fine.
And also I can ping from the lighthouse 1 to client 1.

But we can't connect the nodes each other. It just shows 'handshake timeout' issue for ping.
So if I ping from client 1 to client 2, it can't reach out.

I'm not sure if this is related to our network structure or maybe we missed something?
Hope your reply.
Thanks.

Logs from affected hosts

No response

Config files from affected hosts

Here is the config files

Lighthouse

pki:
  ca: ./ca.crt
  cert: ./lighthouse-1.crt
  key: ./lighthouse-1.key

static_host_map:
  "10.89.12.1": ["srv.******.com:4242"]
  "10.89.12.2": ["srv1.******.com:4242"]

lighthouse:
  am_lighthouse: true
  interval: 60
  hosts:

listen:
  host: 0.0.0.0
  port: 4242

punchy:
  punch: true

relay:
  am_relay: false
  use_relays: true

tun:
  disabled: false  
  dev: nebula1  
  drop_local_broadcast: false  
  drop_multicast: false  
  tx_queue: 500  
  mtu: 1300
  routes:
  
  unsafe_routes:

logging:
  level: info
  format: text

firewall:  
  outbound_action: reject
  inbound_action: reject

  conntrack:
    tcp_timeout: 12m
    udp_timeout: 3m
    default_timeout: 10m
	
  outbound:
    - port: any
      proto: any
      host: any

  inbound:
    - port: any
      proto: any
      host: any

Client

pki:
  ca: ./ca.crt
  cert: ./client-1.crt
  key: ./client-2.key

static_host_map:
  "10.89.12.1": ["srv.******.com:4242"]
  "10.89.12.2": ["srv1.******.com:4242"]

lighthouse:
  am_lighthouse: false
  interval: 60
  hosts:
    - "10.89.12.1"
    - "10.89.12.2"

listen:
  host: 0.0.0.0
  port: 4242

punchy:
  punch: true

relay:
  am_relay: false
  use_relays: true

tun:
  disabled: false  
  dev: nebula1  
  drop_local_broadcast: false  
  drop_multicast: false  
  tx_queue: 500  
  mtu: 1300
  routes:
  
  unsafe_routes:

logging:
  level: info
  format: text

firewall:  
  outbound_action: reject
  inbound_action: reject

  conntrack:
    tcp_timeout: 12m
    udp_timeout: 3m
    default_timeout: 10m
	
  outbound:
    - port: any
      proto: any
      host: any

  inbound:
    - port: any
      proto: any
      host: any

[John Maguire]

Hi @peter0322 - please provide logs as requested by the issue template. Without them, it's very difficult to guess what's happening.

You can try running the following commands to see if your clients are double-NAT'd. If they are, you will likely need to enable the relays functionality of Nebula. This feature can solve many tricky connection scenarios by allowing your relays (oftentimes people use the lighthouses as relays) to relay traffic between the two nodes that are struggling to connect directly.

% go install github.com/pion/stun/cmd/stun-nat-behaviour@latest
% stun-nat-behaviour 2>&1 | grep "NAT mapping"

See this comment to learn how to interpret the results.

[peter0322]

Hi @johnmaguire
Thanks for the fast response.
Let me check.

[peter0322]

Great help @johnmaguire

Now the p2p connection works between nodes with relay option.

Thanks for your help.

[John Maguire]

Glad it was an easy one! :-)