Letsencrypt auto-renewal appears to have broken my proxy-ports settings.
cooperaj opened this issue · comments
Description of problem
At midnight last night I got a notification that my service was down. I investigated as was seeing these errors in the log:
2023/01/13 00:00:41 [error] 1988130#1988130: *2599409 SSL_do_handshake() failed (SSL: error:0A00010B:SSL routines::wrong version number) while SSL handshaking to upstream, client: 31.132.6.98, server: social.n8e.dev, request: "POST /inbox HTTP/2.0", upstream: "https://172.17.0.14:80/inbox", host: "social.n8e.dev"
This pointed me at my port maps being incorrect. True enough they were now set to http:80:80 https:443:80 instead of the functioning http:80:80 https:443:443 that they were originally set to.
I have not got auto updates enabled on the server so it took me a while to find out what happens at midnight. Turns out its the letsencrypt auto-renew cronjob that the plugin uses. It also happened to renew this domain at midnight last night.
=====> Enabling letsencrypt for social.n8e.dev
-----> Enabling ACME proxy for social.n8e.dev...
-----> Getting letsencrypt certificate for social.n8e.dev...
- Domain 'social.n8e.dev'
2023/01/13 00:00:25 [INFO] [social.n8e.dev] acme: Obtaining bundled SAN certificate
2023/01/13 00:00:26 [INFO] [social.n8e.dev] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/194708765317
2023/01/13 00:00:26 [INFO] [social.n8e.dev] acme: Could not find solver for: tls-alpn-01
2023/01/13 00:00:26 [INFO] [social.n8e.dev] acme: use http-01 solver
2023/01/13 00:00:26 [INFO] [social.n8e.dev] acme: Trying to solve HTTP-01
2023/01/13 00:00:27 [INFO] [social.n8e.dev] Served key authentication
2023/01/13 00:00:27 [INFO] [social.n8e.dev] Served key authentication
2023/01/13 00:00:27 [INFO] [social.n8e.dev] Served key authentication
2023/01/13 00:00:32 [INFO] [social.n8e.dev] The server validated our request
2023/01/13 00:00:32 [INFO] [social.n8e.dev] acme: Validations succeeded; requesting certificates
2023/01/13 00:00:33 [INFO] [social.n8e.dev] Server responded with a certificate.
-----> Certificate retrieved successfully.
-----> Installing let's encrypt certificates
-----> Unsetting DOKKU_PROXY_PORT
-----> Unsetting DOKKU_PROXY_SSL_PORT
-----> Setting config vars
DOKKU_PROXY_PORT_MAP: http:80:80
-----> Setting config vars
DOKKU_PROXY_PORT_MAP: http:80:80 https:443:80
-----> Overriding default nginx.conf with detected nginx.conf.sigil
-----> Configuring social.n8e.dev...(using app-supplied template)
-----> Creating https nginx.conf
Enabling HSTS
Reloading nginx
-----> Overriding default nginx.conf with detected nginx.conf.sigil
-----> Configuring social.n8e.dev...(using app-supplied template)
-----> Creating https nginx.conf
Enabling HSTS
Reloading nginx
-----> Disabling ACME proxy for social.n8e.dev...
-----> Done
I've got no idea why it does the port map setting, nor why it does it twice. I also don't know why it regens and reloads nginx twice. Either way the configurations it used both times were not correct.
Steps to reproduce
Run letsencrypt plugin and have it renew a domain that has the port map http:80:80 https:443:443
dokku report $APP_NAME
-----> uname: Linux n8e 5.15.0-56-generic dokku/dokku#62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
-----> memory:
total used free shared buff/cache available
Mem: 3923 2384 167 151 1372 1123
Swap: 7999 1790 6209
-----> docker version:
Client: Docker Engine - Community
Version: 20.10.22
API version: 1.41
Go version: go1.18.9
Git commit: 3a2c30b
Built: Thu Dec 15 22:28:04 2022
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.22
API version: 1.41 (minimum version 1.12)
Go version: go1.18.9
Git commit: 42c8b31
Built: Thu Dec 15 22:25:49 2022
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.14
GitCommit: 9ba4b250366a5ddde94bb7c9d1def331423aa323
runc:
Version: 1.1.4
GitCommit: v1.1.4-0-g5fd4c4d
docker-init:
Version: 0.19.0
GitCommit: de40ad0
-----> docker daemon info:
Client:
Context: default
Debug Mode: true
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Docker Buildx (Docker Inc., v0.9.1-docker)
compose: Docker Compose (Docker Inc., v2.14.1)
scan: Docker Scan (Docker Inc., v0.23.0)
Server:
Containers: 23
Running: 18
Paused: 0
Stopped: 5
Images: 219
Server Version: 20.10.22
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local local-persist
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9ba4b250366a5ddde94bb7c9d1def331423aa323
runc version: v1.1.4-0-g5fd4c4d
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: default
cgroupns
Kernel Version: 5.15.0-56-generic
Operating System: Ubuntu 22.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.832GiB
Name: n8e
ID: CIWV:QUKD:AS4D:VIBI:7UCX:ZGG7:CVGQ:LWOU:NMUR:MRTV:XJNX:SUG4
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
-----> git version: git version 2.34.1
-----> sigil version: 0.9.0build+bc921b7
-----> herokuish version:
herokuish: v0.5.40
buildpacks:
heroku-buildpack-multi v1.2.0
heroku-buildpack-ruby v244
heroku-buildpack-nodejs v202
heroku-buildpack-clojure v90
heroku-buildpack-python v223
heroku-buildpack-java v72
heroku-buildpack-gradle v38
heroku-buildpack-scala v96
heroku-buildpack-play v26
heroku-buildpack-php v227
heroku-buildpack-go v169
heroku-buildpack-nginx v22
buildpack-null v3
-----> dokku version: dokku version 0.29.3
-----> plugn version: plugn: 0.12.0build+3a27594
-----> dokku plugins:
00_dokku-standard 0.29.3 enabled dokku core standard plugin
20_events 0.29.3 enabled dokku core events logging plugin
app-json 0.29.3 enabled dokku core app-json plugin
apps 0.29.3 enabled dokku core apps plugin
builder 0.29.3 enabled dokku core builder plugin
builder-dockerfile 0.29.3 enabled dokku core builder-dockerfile plugin
builder-herokuish 0.29.3 enabled dokku core builder-herokuish plugin
builder-lambda 0.29.3 enabled dokku core builder-lambda plugin
builder-null 0.29.3 enabled dokku core builder-null plugin
builder-pack 0.29.3 enabled dokku core builder-pack plugin
buildpacks 0.29.3 enabled dokku core buildpacks plugin
caddy-vhosts 0.29.3 enabled dokku core caddy-vhosts plugin
certs 0.29.3 enabled dokku core certificate management plugin
checks 0.29.3 enabled dokku core checks plugin
common 0.29.3 enabled dokku core common plugin
config 0.29.3 enabled dokku core config plugin
cron 0.29.3 enabled dokku core cron plugin
docker-options 0.29.3 enabled dokku core docker-options plugin
domains 0.29.3 enabled dokku core domains plugin
enter 0.29.3 enabled dokku core enter plugin
git 0.29.3 enabled dokku core git plugin
letsencrypt 0.18.1 enabled Automated installation of let's encrypt TLS certificates
logs 0.29.3 enabled dokku core logs plugin
mongo 1.23.0 enabled dokku mongo service plugin
network 0.29.3 enabled dokku core network plugin
nginx-vhosts 0.29.3 enabled dokku core nginx-vhosts plugin
plugin 0.29.3 enabled dokku core plugin plugin
postgres 1.26.0 enabled dokku postgres service plugin
proxy 0.29.3 enabled dokku core proxy plugin
ps 0.29.3 enabled dokku core ps plugin
redirect 0.7.1 enabled Plugin for managing application redirects
redis 1.26.2 enabled dokku redis service plugin
registry 0.29.3 enabled dokku core registry plugin
repo 0.29.3 enabled dokku core repo plugin
resource 0.29.3 enabled dokku core resource plugin
run 0.29.3 enabled dokku core run plugin
scheduler 0.29.3 enabled dokku core scheduler plugin
scheduler-docker-local 0.29.3 enabled dokku core scheduler-docker-local plugin
scheduler-null 0.29.3 enabled dokku core scheduler-null plugin
shell 0.29.3 enabled dokku core shell plugin
ssh-keys 0.29.3 enabled dokku core ssh-keys plugin
storage 0.29.3 enabled dokku core storage plugin
trace 0.29.3 enabled dokku core trace plugin
traefik-vhosts 0.29.3 enabled dokku core traefik-vhosts plugin
=====> social.n8e.dev app-json information
App json computed selected: app.json
App json global selected: app.json
App json selected:
=====> social.n8e.dev app information
App created at: 1670442967
App deploy source: social.n8e.dev
App deploy source metadata: social.n8e.dev
App dir: /home/dokku/social.n8e.dev
App locked: false
=====> social.n8e.dev builder information
Builder build dir:
Builder computed build dir:
Builder computed selected:
Builder global build dir:
Builder global selected:
Builder selected:
=====> social.n8e.dev builder-dockerfile information
Builder dockerfile computed dockerfile path: Dockerfile
Builder dockerfile global dockerfile path: Dockerfile
Builder dockerfile dockerfile path:
=====> social.n8e.dev builder-lambda information
Builder lambda computed lambdayml path: lambda.yml
Builder lambda global lambdayml path: lambda.yml
Builder lambda lambdayml path:
=====> social.n8e.dev builder-pack information
Builder pack computed projecttoml path: project.toml
Builder pack global projecttoml path: project.toml
Builder pack projecttoml path:
=====> social.n8e.dev buildpacks information
Buildpacks computed stack: gliderlabs/herokuish:latest-20
Buildpacks global stack:
Buildpacks list:
Buildpacks stack:
=====> social.n8e.dev ssl information
Ssl dir: /home/dokku/social.n8e.dev/tls
Ssl enabled: true
Ssl hostnames: social.n8e.dev
Ssl expires at: Apr 12 23:00:31 2023 GMT
Ssl issuer: C = US, O = Let's Encrypt, CN = R3
Ssl starts at: Jan 12 23:00:32 2023 GMT
Ssl subject: subject=CN = social.n8e.dev
Ssl verified: self signed
=====> social.n8e.dev checks information
Checks disabled list: none
Checks skipped list: none
Checks computed wait to retire: 60
Checks global wait to retire: 60
Checks wait to retire:
=====> social.n8e.dev cron information
Cron task count: 0
=====> social.n8e.dev docker options information
Docker options build: --link dokku.postgres.socialn8edev:dokku-postgres-socialn8edev --link dokku.redis.socialn8edev:dokku-redis-socialn8edev
Docker options deploy: --link dokku.postgres.socialn8edev:dokku-postgres-socialn8edev --link dokku.redis.socialn8edev:dokku-redis-socialn8edev --restart=on-failure:10 -v /var/lib/dokku/data/storage/mastodon:/config
Docker options run: --link dokku.postgres.socialn8edev:dokku-postgres-socialn8edev --link dokku.redis.socialn8edev:dokku-redis-socialn8edev -v /var/lib/dokku/data/storage/mastodon:/config
=====> social.n8e.dev domains information
Domains app enabled: true
Domains app vhosts: social.n8e.dev
Domains global enabled: true
Domains global vhosts: n8e.dev
=====> social.n8e.dev git information
Git deploy branch: main
Git global deploy branch: master
Git keep git dir: false
Git rev env var: GIT_REV
Git sha:
Git source image:
Git last updated at: 1671118869
=====> social.n8e.dev letsencrypt information
Letsencrypt active: true
Letsencrypt autorenew: true
Letsencrypt email: webmaster@networkpie.co.uk
Letsencrypt expiration: 1681340431
=====> social.n8e.dev logs information
Logs computed max size: 10m
Logs global max size: 10m
Logs global vector sink:
Logs max size:
Logs vector sink:
=====> social.n8e.dev network information
Network attach post create:
Network attach post deploy:
Network bind all interfaces: false
Network computed attach post create:
Network computed attach post deploy:
Network computed bind all interfaces: false
Network computed initial network:
Network computed tld:
Network global attach post create:
Network global attach post deploy:
Network global bind all interfaces: false
Network global initial network:
Network global tld:
Network initial network:
Network static web listener:
Network tld:
Network web listeners: 172.17.0.14:443
=====> social.n8e.dev nginx information
Nginx access log format:
Nginx access log path: /var/log/nginx/social.n8e.dev-access.log
Nginx bind address ipv4:
Nginx bind address ipv6: ::
Nginx client max body size: 50m
Nginx disable custom config: false
Nginx error log path: /var/log/nginx/social.n8e.dev-error.log
Nginx global hsts: true
Nginx computed hsts: true
Nginx hsts:
Nginx hsts include subdomains: true
Nginx hsts max age: 15724800
Nginx hsts preload: false
Nginx computed nginx conf sigil path: nginx.conf.sigil
Nginx global nginx conf sigil path: nginx.conf.sigil
Nginx nginx conf sigil path:
Nginx proxy buffer size: 4096
Nginx proxy buffering: on
Nginx proxy buffers: 8 4096
Nginx proxy busy buffers size: 8192
Nginx proxy read timeout: 60s
Nginx last visited at: 1673626009
Nginx x forwarded for value: $remote_addr
Nginx x forwarded port value: $server_port
Nginx x forwarded proto value: $scheme
Nginx x forwarded ssl:
=====> social.n8e.dev proxy information
Proxy enabled: true
Proxy port map: http:80:80 https:443:443
Proxy type: nginx
=====> social.n8e.dev ps information
Deployed: true
Processes: 1
Ps can scale: true
Ps computed procfile path: Procfile
Ps global procfile path: Procfile
Ps procfile path:
Ps restart policy: on-failure:10
Restore: true
Running: true
Status web 1: running (CID: c59b4620a82)
=====> social.n8e.dev registry information
Registry computed image repo: dokku/social.n8e.dev
Registry computed push on release: false
Registry computed server:
Registry global push on release:
Registry global server:
Registry image repo:
Registry push on release:
Registry server:
Registry tag version:
=====> social.n8e.dev resource information
=====> social.n8e.dev scheduler information
Scheduler computed selected: docker-local
Scheduler global selected: docker-local
Scheduler selected:
=====> social.n8e.dev scheduler-docker-local information
Scheduler docker local disable chown:
Scheduler docker local init process: true
Scheduler docker local parallel schedule count:
=====> social.n8e.dev storage information
Storage build mounts:
Storage deploy mounts: -v /var/lib/dokku/data/storage/mastodon:/config
Storage run mounts: -v /var/lib/dokku/data/storage/mastodon:/config
Additional information
No response
Output of failing commands after running: dokku trace:off
No response
Output of failing commands after running: dokku trace:on
No response
See referenced issue - ran into the same myself. Happy to help sponsor!
Dokku itself is what is remapping these ports when we add a certificate. It will take any port http:80 mappings and use the container port to make https:443 mappings.
I'd like to know a bit more about the use case here for not doing that. What sort of apps are ya'll running where 80 and 443 should be passed to different ports on the backend, and why can't they be the same port? I would at least need a test case to figure out how to replicate this.
Okay its mastadon doing something weird.
I think maybe in this case, Dokku should probably just ignore adding any https ports if there are already https:443 port mappings to begin with. Dokku can respect that opinion and log a warning saying it isn't changing ports, which should be fine for cases like this.
To be clear, we still would like dokku to do SSL termination, we just need it to also communicate with the container over 443 - which it does when you initially configure it, but then letsencrypt runs and removes that mapping.
So the initial mapping is what in your case?
Ah you specified at the top.
Basically if we see an https:443 mapping, don't touch the port mappings. I think thats what I'll change it to.