Allow real .dev DNS resolution alongside support for local .dev domains
kapowaz opened this issue · comments
At Deliveroo we use puma-dev extensively for local development, and for historical reasons have configured .dev as our domain suffix for numerous separate projects. Now that .dev is a real world TLD, this is proving problematic as it makes any real-world website hosted at a .dev domain completely inaccessible without removing puma-dev.
The recommendation is obviously to move to using .test for local development, and this is naturally something we’re eager to explore fully. However there are a few (significant) problems with this:
- Many existing internal and tools are configured to expect
.devdomains for local development (for referrer URLs for example), so it’ll take quite a while to audit and move these over. - (Most significantly) Google SSO only allows real world domains as the referrer, and so where we use Google SSO for local development, this only works when using
.devdomains.
The latter one seems to be the biggest blocker here — I’ve already reached out to Google to see if they can support non-real world domains in their OAuth configuration and I’m waiting to hear more news, but it’s safe to assume that this isn’t likely to be resolved quickly. It seems like it’s only a quirk of timing that we were ever able to enable Google SSO in our products — in the period post-Google acquiring the .dev TLD it became a real domain (and presumably was being used internally), so it was supported for SSO, even though at that point no real .dev domains existed on the (public) internet.
Whilst we continue to explore a path away from using .dev in local development, I was wondering about an alternative: would it be possible to support a mode in puma-dev which allowed the host system to at least attempt a DNS lookup for a .dev domain, but only if it fails have puma-dev attempt to route to the locally configured app? Or as an even more crude alternative, allow for a configuration where (say) you create a file in ~/.puma-dev with the name of the domain, containing the IP address it should resolve to?
Obviously these are just ideas and they may not be practical or possible to implement, but I’d welcome thoughts and suggestions. We have a number of enthusiastic Go engineers so it’s possible we may even be able to help out by supplying a patch.
I think you should probably just move to using .puma or .pdev or another domain that isn't a real tld. Note that .test SHOULD NOT be used. It does strange things when combined with subdomains and rails.
I don't think puma-dev can solve your problem.
I know that's not what you want to hear. Google buying .dev really sucked for us too. But that ship has sailed and I don't think we can do much about it.
Your only other option is to buy the .dev domains you want and point them to localhost - but note that I think .dev requires tls.
I think maybe puma-dev should log a warning when using .dev and .test now since they were so common but browsers are now doing weird things with them. It's not puma-dev's fault.
My google-fu is letting me down, do you have any more information about why .test should not be used @idyll ? We haven't run into any problems with subdomain .test Rails apps here (yet).
@barrywoolgar I'm seeing cookies being set incorrectly but only with .test domains.
Instead of applying across all subdomains they are only being applied to the bare domain.
For example .myapp.test is being changed to myapp.test.
And only for .test domains.
I have not had time to trace back why.
So I haven't found the time (perhaps I will soon!) but I have gotten puma.dev, the real global domain with the intention of using it as the default domain for puma-dev. Ie, myapp.puma.dev would go to your local machine.
I need to get it wired up still, maybe this is the push I need to do so!
Thanks for the replies all!
I think you should probably just move to using .puma or .pdev or another domain that isn't a real tld. Note that .test SHOULD NOT be used. It does strange things when combined with subdomains and rails.
Unfortunately as I mentioned Google SSO presently requires that the referring domain be a real-world one. So for example, if I’m trying to use Google SSO to log in to https://deliveroo.co.uk.dev/ locally, since uk.dev is a real domain, it will permit the request (even though when it redirects me back, puma-dev intercepts the redirect to deliveroo.co.uk.dev and the SSO auth works). (I’m not familiar with the issue with .test domains, but other than SSO issues, I’ve not experienced any problems using puma-dev with .test locally, thankfully.)
Your only other option is to buy the .dev domains you want and point them to localhost - but note that I think .dev requires tls.
When testing the Deliveroo consumer website our current setup is to append .dev to the real-world domain of the equivalent site, so for e.g. deliveroo.co.uk.dev or deliveroo.fr.dev; unfortunately this would require that we buy uk.dev, fr.dev etc. which are all already owned.
I don't think puma-dev can solve your problem.
I know that's not what you want to hear. Google buying .dev really sucked for us too. But that ship has sailed and I don't think we can do much about it.
Were I familiar enough with Go, I think I’d have a go (ha) at writing the feature I describe myself, as I’m fairly certain it’s possible to have puma-dev attempt a DNS lookup on the requested domain first, and only use the local resolver if it’s a non-existent domain. As I mention, there’s quite a few Go engineers here, so I’m going to talk to more of them about getting their help on this. Obviously this functionality would increase the footprint of puma-dev, and it’d be something only legacy .dev users would ever need. If we submitted a PR to do this, would it be rejected? Or could it be supported as an installation option, that has to be opted into?
If we submitted a PR to do this, would it be rejected?
I can't speak for @evanphx, but for me I think it would be rejected. I think the path forward here is move everybody from legacy .dev domains to .test and puma.dev.
I think the path forward here is move everybody from legacy .dev domains to .test and puma.dev.
Unfortunately neither of those is practical in the medium term, given that we’re not able to influence Google’s roadmap for how they support non-real domains for SSO. Using the puma.dev domain (or subdomains thereof) would also not work for us since it’s a real world domain owned by a third party, and so I think there would be legitimate concerns about security.
If we assume that moving off .dev isn’t feasible for the time being, what suggestions might you have for how we can resolve our particular situation?
What about .test? I know there are issues with .test now but I would like to fix those instead, as .test is guaranteed to never be owned by a third party.
What about .test?
It’s not a real domain, and so Google SSO doesn’t work with it — it literally refuses to honour the callback to any such domain. I’ve reached out to Google about this, but I have no hard dates as to when (or even if) they might ever fix this.
Edit: to clarify, we use Google SSO for authentication across pretty much all of our applications, and so in any app where we use Google SSO in development mode, it needs to be able to route back to the development domain once authentication has succeeded.
YMMV, but I've started using a domain with a real TLD that I'll never visit for real, e.g. loc.al. You can then access your app at http://myapp.loc.al and Google SSO will work correctly locally.
# setup including custom domain
puma-dev -d test:localhost:loc.al -install
sudo puma-dev -d test:localhost:loc.al -setup
# manually alias to TLD and domain
cd ~/.puma-dev
ln -s ~/src/myapp myapp.loc
ln -s ~/src/myapp myappAlternatively, you can use a domain that you own, like myname.io. if you symlink to myapp.myname, puma-dev will resolve correctly.
Kind of annoying is that there's very few TLDs that we know will never be registered...thanks a lot gTLD.
Based on @nonrational’s suggestion, my colleague Arnaud created this script which appears to solve the problem for us. Basically what it does is register explicitly all the real-world TLDs we use (e.g. deliveroo.co.uk.dev) in development, so that actual real-world .dev domains aren’t handled by puma-dev, only these exceptions. Since all other local development domains are subdomains of deliveroo.dev, we can register this as one to handle and it will ‘just work’ for those.
I need to get it wired up still, maybe this is the push I need to do so!
FWIW I got myapp.puma.dev working locally using the current (0.12) release by running ln -s /path/to/myapp ~/.puma-dev/myapp.puma
The solution that works for me has very little to do with puma-dev or golang.
We need a specific list of domains to be resolved and leave the rest untouched.
If puma was configured to use .dev on a Mac by running the command puma-dev -install -d dev,
then do the following to only direct deliveroo.co.uk.dev to puma.
mv /etc/resolver/dev /etc/resolver/deliveroo.co.uk.dev
The other domains can be easily added by copying
cp /etc/resolver/deliveroo.co.uk.dev /etc/resolver/deliveroo.fr.dev
cp /etc/resolver/deliveroo.co.uk.dev /etc/resolver/deliveroo.es.dev
cp /etc/resolver/deliveroo.co.uk.dev /etc/resolver/deliveroo.it.dev
Explanation
when you run:
puma-dev -install -d devpuma creates a file /etc/resolver/dev with this content:
# Generated by puma-dev
nameserver 127.0.0.1
port 9253
which makes MacOS direct all requests to .dev domains to 127.0.0.1:9253 which is puma-dev