Adding AAAA record to DNS makes Let's Encrypt validation fail.
cyplo opened this issue · comments
Heyo, First of all, thank you for an AMAZING project ! :)
I'm trying to migrate from some legacy ball-of-mud mail server config to this.
Everything seems to be working very smoothly, except for the IPv6 Let's Encrypt validation.
By default Let's Encrypt tries to validate domain ownership via IPv6 if there is an AAAA record for a particular domain.
If I set only A records for the mail server - everything works, including Let's Encrypt.
If I add AAAA record before validating the certificate - the Let's Encrypt setup fails.
Hypothesis: nginx not listening on IPv6 ? (haven't checked yet)
SNM Version: 2.1.2
Nixos Version: 17.09
Relevant part of the config to reproduce:
{ config, pkgs, ... }:
{
imports = [
<nixpkgs/nixos/modules/virtualisation/amazon-image.nix>
(builtins.fetchTarball "https://github.com/r-raymond/nixos-mailserver/archive/v2.1.2.tar.gz")
];
ec2.hvm = true;
environment.systemPackages = with pkgs; [ vim tmux fail2ban python ];
services = {
fail2ban = {enable = true;};
};
mailserver = {
enable = true;
fqdn = "mail.test.cyplo.net";
domains = [ "test.cyplo.net" ];
certificateScheme = 3;
enableImap = true;
enableImapSsl = true;
enablePop3 = false;
enablePop3Ssl = false;
enableManageSieve = true;
virusScanning = false;
loginAccounts = {
"cyplo@test.cyplo.net" = {
hashedPassword = "somehash";
aliases = [ "postmaster@test.cyplo.net" "admin@test.cyplo.net" ];
catchAll = [ "test.cyplo.net" ];
};
};
};
}
What I expected to happen:
When I set AAAA dns record to point to the server's IPv6 address I expect Let's encrypt validation to succeed.
What happened:
Let's Encrypt validation fails.
Relevant journald log:
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: 2018-02-25 21:36:59,516:DEBUG:acme.client:622: Sending GET request to https://acme-v01.api.letsencrypt.org/a
cme/authz/4EWNTLcTMtFh-97q3r2iQKaj65NcK-sbWY5BnydO844.
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: 2018-02-25 21:36:59,768:DEBUG:urllib3.connectionpool:396: https://acme-v01.api.letsencrypt.org:443 "GET /acm
e/authz/4EWNTLcTMtFh-97q3r2iQKaj65NcK-sbWY5BnydO844 HTTP/1.1" 200 1508
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: 2018-02-25 21:36:59,769:DEBUG:acme.client:638: Received response:
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: HTTP 200
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: Server: nginx
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: Content-Type: application/json
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: Content-Length: 1508
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: Link: <https://acme-v01.api.letsencrypt.org/acme/new-cert>;rel="next"
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: Replay-Nonce: s4pUa_h9KEbzTPKoCnkMN1KGK7BDsQwsvRnBif1Ddh4
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: X-Frame-Options: DENY
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: Strict-Transport-Security: max-age=604800
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: Expires: Sun, 25 Feb 2018 21:37:00 GMT
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: Cache-Control: max-age=0, no-cache, no-store
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: Pragma: no-cache
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: Date: Sun, 25 Feb 2018 21:37:00 GMT
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: Connection: keep-alive
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: {
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "identifier": {
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "type": "dns",
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "value": "mail.test.cyplo.net"
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: },
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "status": "invalid",
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "expires": "2018-03-04T21:36:49Z",
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "challenges": [
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: {
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "type": "http-01",
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "status": "invalid",
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "error": {
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "type": "urn:acme:error:connection",
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "detail": "Fetching http://mail.test.cyplo.net/.well-known/acme-challenge/D9e0V3DUm33XY-51UU98gSlFgVB3xyIQ4Mhijh8F918: Timeout",
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "status": 400
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: },
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/4EWNTLcTMtFh-97q3r2iQKaj65NcK-sbWY5BnydO844/3599723908",
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "token": "D9e0V3DUm33XY-51UU98gSlFgVB3xyIQ4Mhijh8F918",
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "keyAuthorization": "D9e0V3DUm33XY-51UU98gSlFgVB3xyIQ4Mhijh8F918.c3Fkz57ZRlw9vrxvubrqmb_QU1Bup4hRyC8c11t0IlU",
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "validationRecord": [
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: {
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "url": "http://mail.test.cyplo.net/.well-known/acme-challenge/D9e0V3DUm33XY-51UU98gSlFgVB3xyIQ4Mhijh8F918",
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "hostname": "mail.test.cyplo.net",
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "port": "80",
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "addressesResolved": [
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "18.196.18.46",
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "2a05:d014:36d:a401:f8d1:fdcc:cdee:db7e"
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: ],
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "addressUsed": "2a05:d014:36d:a401:f8d1:fdcc:cdee:db7e"
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: }
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: ]
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: },
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: {
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "type": "dns-01",
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "status": "pending",
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/4EWNTLcTMtFh-97q3r2iQKaj65NcK-sbWY5BnydO844/3599723909",
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "token": "RyhcWj-o5iKVzo6zfsUHwNTTpAYfgH299m0V9IAWJyc"
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: }
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: ],
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "combinations": [
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: [
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: 1
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: ],
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: [
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: 0
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: ]
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: ]
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: }
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: 2018-02-25 21:36:59,770:ERROR:simp_le:1404: CA marked some of the authorizations as invalid, which likely means it could not access http://example.com/.well-known/acme-challenge/X. Did you set correct path in -d example.com:path or --default_root? Are all your domains accessible from the internet? Please check your domains' DNS entries, your host's network/firewall setup and your webserver config. If a domain's DNS entry has both A and AAAA fields set up, some CAs such as Let's Encrypt will perform the challenge validation over IPv6. If you haven't setup correct CAA fields or if your DNS provider does not support CAA, validation attempts after september 8, 2017 will fail. Failing authorizations: https://acme-v01.api.letsencrypt.org/acme/authz/4EWNTLcTMtFh-97q3r2iQKaj65NcK-sbWY5BnydO844
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: 2018-02-25 21:36:59,770:DEBUG:simp_le:1108: Removing validation file at /var/lib/acme/acme-challenge/.well-known/acme-challenge/D9e0V3DUm33XY-51UU98gSlFgVB3xyIQ4Mhijh8F918
Are you sure that Let's Encrypt can get the DNS record for your domain?
dig is telling me that there is no such record:
❯ dig AAAA test.cyplo.net
; <<>> DiG 9.11.2 <<>> AAAA test.cyplo.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24788
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;test.cyplo.net. IN AAAA
;; AUTHORITY SECTION:
cyplo.net. 3594 IN SOA ns-111.awsdns-13.com. admin.cyplo.net. 2018020301 3600 1800 604800 600
;; Query time: 42 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Mon Feb 26 10:46:14 ACDT 2018
;; MSG SIZE rcvd: 105
You can check with this too: https://mxtoolbox.com/SuperTool.aspx
Heyo !
Thank you for checking :) It indeed does not have the record right now, as I deleted it to be able to make some further progress with the config. Looking at the logs pasted above:
"validationRecord": [
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: {
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "url": "http://mail.test.cyplo.net/.well-known/acme-challenge/D9e0V3DUm33XY-51UU98gSlFgVB3xyIQ4Mhijh8F918",
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "hostname": "mail.test.cyplo.net",
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "port": "80",
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "addressesResolved": [
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "18.196.18.46",
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "2a05:d014:36d:a401:f8d1:fdcc:cdee:db7e"
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: ],
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: "addressUsed": "2a05:d014:36d:a401:f8d1:fdcc:cdee:db7e"
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: }
Feb 25 21:36:59 ip-10-0-27-169.eu-central-1.compute.internal acme-mail.test.cyplo.net-start[819]: ]
- I think that it may have resolved the addresses properly on the DNS level it had troubles validating though - am I reading this correctly ?
thanks ! :)
Unfortunately I don't have an idea about your problem. If you remove SNM and add acme manually does it work? Maybe this is a bug in Nixos' acme support?
@cyplo could you share your entire config? AFAICT nginx will listen on [::] if config.networking.enableIPv6 is true, which I don't see in the snippet you posted. https://github.com/NixOS/nixpkgs/blob/93bfd8921fa26bc7046d3cb64c05ffe7f30bb9bf/nixos/modules/services/web-servers/nginx/default.nix#L20
Thank you both for your support :)
I'm going to try both seeing if pure ACME without SNM works and if setting nginx config explicitely helps.
The config I've pasted above is the full config from that machine - I don't have nginx configured separately, as I don't need it on the box. My understanding is that ACME needs it to setup a temporary validation server.
thanks again and will keep you posted :)
Hello !
I've tried different combinations and:
🎉 Resolved and not a bug in SNM 🎉 - was an unrelated network configuration problem - apologies :)
p.s. - I tried with config.networking.enableIPv6 and without it - does not make a difference if the box has an assigned IPv6 address - ACME works either way.
thanks again !