Civo IP mismatch
alexellis opened this issue · comments
It appears that Civo are using some kind of IP mapping, where the inlets PRO installation tries to get the public IP by reaching out to an IP finding service, but the IP that came back was incorrect.
inlets-pro client --url "wss://91.211.153.165:8123/connect" --token "$TOKEN" --license-file ~/LICENSE --upstream http://192.168.0.26 --ports 80,443,8080
2020/09/23 10:08:47 Welcome to inlets-pro! Client version 0.7.0
2020/09/23 10:08:47 Licensed to: Alex Ellis <alex@openfaas.com>, expires: 48 day(s)
2020/09/23 10:08:47 Upstream server: http://192.168.0.26, for ports: 80, 443, 8080
Error: unable to download CA from remote inlets server for auto-tls: Get https://91.211.153.165:8123/.well-known/ca.crt: x509: certificate is valid for 185.136.235.146, not 91.211.153.165$ civo instance list
+--------------------------------------+---------------+----------+-----------+------+----------+--------+----------------+------------+--------+
| ID | Hostname | Size | Cpu Cores | Ram | SSD disk | Region | Public IP | Private IP | Status |
+--------------------------------------+---------------+----------+-----------+------+----------+--------+----------------+------------+--------+
| 0eb29dc5-b2ea-43c6-8bbe-2fd78010be92 | crazy-clarke6 | g2.small | 1 | 2048 | 25 | lon1 | 91.211.153.165 | 10.0.0.20 | ACTIVE |
+--------------------------------------+---------------+----------+-----------+------+----------+--------+----------------+------------+--------+
SSH:
civo@crazy-clarke6:~$ cat /etc/default/inlets-pro
AUTHTOKEN=TOKENVALUE
IP=185.136.235.146
That IP gets there via export IP=$(curl -sfSL https://checkip.amazonaws.com)
Running it later on, interactively gives:
civo@crazy-clarke6:~$ curl -sfSL https://checkip.amazonaws.com
91.211.153.165
civo@crazy-clarke6:~$
Any thoughts @alejandrojnm / @kaihoffman?
Given it a month for people to reply, now trying another one of the team.. pinging @andyjeffries
Andy, is there any reason the public IP would report as "wrong" during the Civo cloud-init phase?
I believe it goes via a router's IP address, so that may not work. A guaranteed way of it working (at least until Kube100 Platform goes live) is to use OpenStack's metadata service:
curl http://169.254.169.254/2009-04-04/meta-data/public-ipv4
Out of interest, how do you get /etc/default/inlets-pro? Is that using the same check-up.amazonaws.com service?
It's generated here with a similar approach for both types of inlets. https://github.com/inlets/inletsctl/blob/master/pkg/provision/userdata.go
Hmmm, so either way it uses checkip.amazonaws.com? That's weird then that it changes; maybe it uses a router's IP up until the public IP is connected. The only thing I can think is to use the official meta-data service, but I'm not sure how you'd check for a failure first in order to try that, because the previous IPs seem to be valid.
I wondered if you run user-data in the traditional way, or whether it's run as part of the image/instance builder? When we spoke last year, I'd requested this feature and at the time I think it had to be implemented in a non-conventional way.
The alternative is a switch statement just for the Civo provider, that gets the IP and generates /etc/default/inlets-pro upon first boot.
Thanks again for taking a look at this.
The user-data is run during cloud-init, but we already use it to initialise the instance. So the way we handled this is to allow the users to specify a custom script (bash, etc) that is run as part of the cloud-init. The API and CLI support this. I don't know if that helps?
The switch statement could be for all OpenStack providers (although I don't know if you provide adaptors/plugins/etc for any others)?
@dirien just ran into this again.
@andyjeffries how would you recommend we obtain the instance IP for Civo?
curl ifconfig.me works nicely from that list I believe, returns just the IP address with no trailing newline.
The mechanism that we are using works well on all other providers, and isn't dissimilar to what you've shared. I think the issue is that the way cloud-init is run / built, means it runs on another machine than the actual host. Am I roughly right there?
During cloud-init build / phase -> the IP is wrong for the instance
During an SSH session after cloud-init -> the IP is correct
This was explained in the thread above.
Cloud-init is generated in our Operator, but run on the VM itself. The IP is known before the instance starts building. We could try to inject the public IP in to a standard file on the filesystem, however, it means if we move the IP to a different instance in the future, that file would be out of date. So I think this is the safest way to do it.
I don't know why it's not working for you...
I tried to look at https://github.com/inlets/inletsctl/blob/master/pkg/provision/userdata.go (to see how you're doing it, from the thread above) but it 404s.
OpenStack may be doing it differently though, my comments above only refer to the new CivoStack regions.
@andyjeffries -> https://github.com/inlets/inletsctl/blob/master/cmd/create.go#L540
Should show you the userdata aka cloud-init
Thank you @dirien . So if you run this on Civo, what IP address do you get at that point and could you leave the instance running (and let me know it's ID) and I'll try to debug what's happening.
I believe that Dinesh worked with @dirien and this is all resolved now.
I think we can close thos one.