Peerage.Via.Dns not working with Kubernetes headless service

[Christopher J. Bottaro]

Easiest to show by pasting output of some commands...

$ kubectl logs channel-app-web-79865c8768-8md9p
03:07:09.586 [debug] [Peerage 1.0.3][ Elixir.Peerage.Via.Dns] Discovery every 10s.

     NAME                RESULT OF ATTEMPT

     LIVE NODES
     web@100.96.5.189 (self)

$ kubectl logs channel-app-web-79865c8768-gsrq
...
03:09:05.109 [debug] [Peerage 1.0.3][ Elixir.Peerage.Via.Dns] Discovery every 10s.

     NAME                RESULT OF ATTEMPT

     LIVE NODES
     web@100.96.5.188 (self)

$ kubectl exec channel-app-web-79865c8768-8md9p -- cat apps/web/config/prod.exs
...
config :peerage, via: Peerage.Via.Dns,
  dns_name: "channel-app",
  app_name: "web"

$ kubectl exec channel-app-web-79865c8768-8md9p -- host channel-app
channel-app.production.svc.cluster.local has address 100.96.5.189
channel-app.production.svc.cluster.local has address 100.96.5.188

Both pods are using the same cookie and both have containerPort: 4369 set.

What am I doing wrong?

[Luc Fueston]

That's enough for a place to start -- I didn't call this out much until a bit further down in the readme, but here, under the 'longer example', there's something that might be relevant:

Your config/prod.exs might look like this:

config :peerage, via: Peerage.Via.Dns,
dns_name: "myapp", # or k8s FQDN: "myapp.ns.svc.clust.local"
app_name: "myapp"

The 'FQDN' comment; I'd try changing the dns_name to the fully qualified name.

(If that doesn't work, next crude debug step might be to use a shell from within the same image, and do dig/nslookup to confirm the names that can be resolved).

[Luc Fueston]

Ah, sorry, quick response missed that you ran host.

Righto, next keyword I'd look into -- inet_dist_listen_max, setting min/max to same port in your vm.args (like port 9100), and then making sure that one is also exposed in container image and in k8s containerPort. (sorry that these notes are a bit disjointed, but if the above is it, hopefully that might be enough to point in right direction).

[Christopher J. Bottaro]

Using FQDN worked! Thank you.

Side note, when I did nslookup channel-app (not using the FQDN) it gives the correct addresses, but as a Non-authoritative answer. I don't know much about DNS, so not sure why or how that matters.

# nslookup channel-app
Server:		100.64.0.10
Address:	100.64.0.10#53

Non-authoritative answer:
Name:	channel-app.production.svc.cluster.local
Address: 100.96.5.196
Name:	channel-app.production.svc.cluster.local
Address: 100.96.5.195