Question re external-dns
andye2004 opened this issue · comments
Firstly, apologies if I am missing something really simple here.
We are testing Istio and what we want is to update pfsense DNS entries using Istio Gateway resource(s), as opposed to using standard k8s Ingress resources. This is supported by external-dns but looking through the list of providers it looks like only RFC2136 is suitable for use with pfsense. The team that manages the pfsense appliances are reluctant to set-up RFC2136 and so we need to find another solution, this is what has brought me here :)
Reading the intro to this project there is mention of using kpc in conjunction with external-dns but I can't find any documentation of how this would be achieved. Does one exist? Could you point me at it?
If not, would it be possible to provide some bullet points to give me a head start?
Many thanks!
Hey thanks for the interest! So I only mentioned that to say they could be used simultaneously. There is no direct integration/connection between the two projects.
So, help me understand your scenario a bit more. You want gateway resources to update the DNS config (unbound or dnsmasq) in pfSense as if you were entering the entries manually via the UI yeah?
Wow, thanks for such a quick response!
My experience with pfsense is pretty much non-existent and I've never looked at the UI but I guess what you've suggested is essentially what I want to do except that our instance is ONLY running dnsmasq.
In essence, I want to create probably an A record that will point to an Istio ingress gateway running inside our on-prem cluster, this ingress gateway will then lb/route traffic inside the cluster based on service name. The ingress gateway is a service type: LoadBalancer and it's external IP is set via MetalLB. As and when new services are created inside the cluster I want to associate the new service with the external IP address of the Istio ingress gateway and then upsert an entry in pfsense dnsmasq.
This will mean that dns resolution of any service within the cluster should always resolve to the same IP address, e.g. the ingress gateway, and standard host/SNI will then come into play inside the Istio service mesh for routing etc.
Yeah ok, only dnsmasq is fine. It's setup to allow making entries in one or both of the DNS implementations.
I don't have much experience with istio so I'll have to learn that a little closer but this is (as you mentioned) essentially how the ingress solution works. Making new DNS alias entries for each new ingress.
Let me read up a little on istio gateway and if you don't mind send over an example config of both the gateway and a service and we'll see what we can do.
Assuming there's a sane way to map DNS entries to IPs it shouldn't be too bad to implement. The code for DNS is pretty abstract so adding new DNS plugins usually is a pretty minimal exercise.
Correction, I forgot the ingress plugin actually creates new entries not alias entries. It uses the status block on the kubernetes resource. Do those istio resources have a status block after they're created with remotely helpful data?
Thanks again for such a speedy follow up. I've had a chat with a senior person on the team that looks after our pfsense instance and he says there would be no chance that direct, programmatic modification of dnsmasq entries by a third party service or software would be allowed without considerable vetting of said service / software so it seems that even this route is blocked for us.
He did suggest an alternative to me that I will spend a day or two looking into but it is definitely a feasible , and possibly even more desirable, solution that would mean we could use external-dns and just setup a zone in dnsmasq that routes requests for a new domain to an independent DNS server.
Ok cool. Keep me updated. I completely understand the reluctance to use such a tool.
I dug through the istio resources I have in a test cluster and did not see a status block on the resources. I could be missing something but didn't find anything meaningful. I'd have to read through the external-dns code to see how they're gathering the list of hosts but from what I can tell they are monitoring the hosts array on the Gateway objects. Not sure if that's sane or not and/or if VirtualService objects would come into play or not. I'd be interested in knowing the 'proper' location to discover the mappings.
In any case, sounds like you're on track for a good solution for your use-case. We have (and I've had reports from many others) been using this little controller in production environments for a bit over a year with 0 issues/explosions/whatever so if you do need to revisit you can say "hey, but the guy says it works for him" to your senior admin :) In all seriousness, we run it in conjunction with gitlab review apps as part of ci so a fair amount of churn is going on all the time (for the various plugins/pieces) and it's never puked on us or eaten the config thank goodness.
Your reading of Istio is pretty spot on, there is an istio-ingressgateway (using envoy under the covers) that acts as an API gateway - basically it acts as a reverse proxy for services running inside the cluster.
From an envoy perspective, proxying is (in VERY broad terms) broken into three parts, receiving, filtering (which includes routing), and forwarding. Receiving is obviously envoy opening server ports for inbound messaging; forwarding is... well... forwarding requests, e.g. sending to upstream services via a configuration block called a cluster. Routing is effectively the glue between the receiving port and forwarding cluster, routing is important as a single receiving server port can be used to proxy multiple upstream services. All of this configuration can be dynamically altered at runtime via grpc APIs.
Getting back to Istio, at start-up, the ingressgateway knows nothing about the services that it should be proxying, e.g. what ports to open, which clusters to forward on to, or how to route between them. It discovers this via it's grpc APIs as and when Gateways and VirtualServices definitions are applied to the cluster.
A Gateway definition is used to configure the ingressgateway to bind/listen on a specified port, if the port is already bound (e.g. another service has already configured that port) then obviously no binding will take place. The Gateway definition can also add other details like what domains are available on the specified port, e.g www.example.com, www.example.org, whether TLS should used, what certificate(s) should be served on the port etc.
A VirtualService provides the routing information and upstream service definitions in a single k8s resource file, and in the same way that a single Gateway resource can specify multiple receiving services, the VirtualService resource can specify multiple forwarding destinations. The VirtualService also specifies how incoming messages can be mapped from the receiving port to the forwarding cluster. Generally speaking, a VirtualService will point at one or more k8s Service definitions using one or more match blocks containing a spec.http.route.destination.host attribute that specifies the name of the service, e.g. service-name.namespace.svc.cluster.local, or if the VirtualService is in the same namespace as the Service, then simply via the service short name.
So, regarding your question about the sanity of using the Gateway instance to create the DNS entry, I think that is probably the correct thing to do. The fact that a service may or may not be running at the endpoint is irrelevant to DNS resolution.
Of course, for this to work simply off the Gateway resource requires that the host field actually contains a valid DNS name, if it contains only namespace mappings or wildcard entries (which it can) then there can't be any identifiable service to register with the DNS server. As you noted, looking through the code for external-dns Istio plugin might yield some more useful information but there is nothing obvious from interrogating these resources on my cluster.
NOTE: Out of the box the Istio ingressgateway is configured with a number of ports and a loadbalancing service is created by default but there is no reason that this needs to be the case. The ingressgateway can be deployed without any ports and without a Service as outlined above and configured later.
Apologies for such a long answer but hopefully it will provide some context that the docs don't readily provide.
Thanks for the great explanation! In essence what's confusing is that hosts are defined in both the gateway and VS so from a mapping perspective I'm not sure what would make the most sense.
Can a VS have a host which isn't defined in the gateway? I'd assume the would only be relevant if the gateway was configured with a wildcard?
Can a VS have a host which isn't defined in the gateway? I'd assume the would only be relevant if the gateway was configured with a wildcard?
Correct on both accounts.
Yeah, seems like the only sane way to deal with it would be to use the gateway. Also it seems like a narrow window where people aren't just using a wildcard on the gateway but I could be way off. Managing those hosts manually seems like a headache unless something is automatically updating the resource (doubtful).
Just by way of a follow-up, I've actually got everything working now using external-dns sending updates to a secondary PowerDNS service acting as an authority on specific zones. Our corporate pfsense instance is still waiting to have the specific zones/forwarding added now but everything I need to do is working very nicely.
Main reason for the follow up is to say that I am using the external-dns Istio plugin making updates from Gateway instances as previously discussed, but after your last comment I wanted to highlight something. In the configuration for external-dns there is a domain-filter that once set filters out anything not in a specific domain, or sub-domain. What this means, for me anyway, is that the worry around wildcards is reduced. Just thought it worth mentioning.
Thanks again for speedy follow-ups to my questions etc, if you do decide to look into this a bit more and have a specific question I'm happy to help - if I can of course - I'll keep following this issue just in case.
Thanks for the follow-up!
Yeah, I mostly meant it would be a pain to not use wildcards in the gateway, not that the pain would be filtering/limiting valid stuff with the controller per-se (I generally allow that with the other DNS plugins already via regex).
In any case thanks for sharing your comments and thoughts with me! It's helped me understand istio a little better. I've avoided mostly because I need support for ingress objects and so haven't dug into it much.
I will make note, if you need powerful auth another project of mine does work with isito/envoy: https://github.com/travisghansen/external-auth-server