LetsEncrypt inside Docker; `502` error.
c4lliope opened this issue · comments
Description of problem
A 502 error upon dokku letsencrypt:enable #{app},
meaning the ACME process ends up issuing a 403 error.
Logs included herein.
Reproduce
on a Fedora CoreOS machine, on an arm64 processor.
- Add a custom plugin in
/var/lib/dokku/plugin-listto repair #235:
letsencrypt: https://base.assembled.app/code/dokku-letsencrypt.git
# or
letsencrypt: https://github.com/c4lliope/dokku-letsencrypt.git- Deploy dokku inside docker; the command I am using is:
#!/bin/bash
set -e
sudo docker container run \
--privileged \
--env DOKKU_HOSTNAME=silicon.am \
--env DOKKU_HOST_ROOT=/var/lib/dokku/home/dokku \
--env PERMISSIONS_CALLS='--privileged' \
--name dokku \
--publish 3022:22 \
--publish 80:80 \
--publish 443:443 \
--volume /var/lib/dokku:/mnt/dokku \
--volume /var/run/docker.sock:/var/run/docker.sock \
-d \
dokku/dokku:latest- Add SSH keys, and configure
~/.ssh/configto use port 3022. - Build and deploy an application, and enable
letsencrypt.
alias dokku='sudo docker exec -it dokku dokku'
APP=check
DOMAIN=silicon.am
dokku config:set --global DOKKU_LETSENCRYPT_EMAIL='...'
dokku apps:create $APP
dokku git:initialize $APP
dokku domains:set $APP $APP.$DOMAIN
# deploy application by `git push` from local machine.
dokku letsencrypt:enable $APPActual Results
[core@localhost ~]$ dokku letsencrypt:enable check
=====> Enabling letsencrypt for check
-----> Enabling ACME proxy for check...
ok: run: nginx: (pid 5728) 4189s
-----> Getting letsencrypt certificate for check...
- Domain 'check.silicon.am'
2022/07/11 20:48:29 [INFO] [check.silicon.am] acme: Obtaining bundled SAN certificate
2022/07/11 20:48:29 [INFO] [check.silicon.am] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/129493634526
2022/07/11 20:48:29 [INFO] [check.silicon.am] acme: Could not find solver for: tls-alpn-01
2022/07/11 20:48:29 [INFO] [check.silicon.am] acme: use http-01 solver
2022/07/11 20:48:29 [INFO] [check.silicon.am] acme: Trying to solve HTTP-01
2022/07/11 20:48:35 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/129493634526
2022/07/11 20:48:35 Could not obtain certificates:
error: one or more domains had a problem:
[check.silicon.am] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: 98.218.220.151: Invalid response from http://check.silicon.am/.well-known/acme-challenge/2J5QPzmJ6-Y9iksVf2LhpzmtJAoGaXbEZKY1Yz-dWFs: 502
-----> Certificate retrieval failed!
! Failed to setup letsencrypt
! Check log output for further information on failure
[core@localhost ~]$ Expected Results
SSL should be enabled on application's domains.
Environment Information
dokku report APP_NAME
[core@localhost ~]$ dokku report check
-----> uname: Linux aa784c360d43 5.18.5-200.fc36.aarch64 #1 SMP PREEMPT_DYNAMIC Thu Jun 16 14:28:32 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
-----> memory:
total used free shared buff/cache available
Mem: 1878 544 560 5 773 1240
Swap: 0 0 0
-----> docker version:
Client:
Version: 20.10.12
API version: 1.41
Go version: go1.16.2
Git commit: 20.10.12-0ubuntu2~20.04.1
Built: Wed Apr 6 02:16:12 2022
OS/Arch: linux/arm64
Context: default
Experimental: true
Server:
Engine:
Version: 20.10.17
API version: 1.41 (minimum version 1.12)
Go version: go1.18.2
Git commit: f756502
Built: Sat Jun 11 05:01:17 2022
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: 1.6.6
GitCommit:
runc:
Version: 1.1.1
GitCommit: e910b9f-dirty
docker-init:
Version: 0.19.0
GitCommit:
-----> docker daemon info:
Client:
Context: default
Debug Mode: true
Server:
Containers: 2
Running: 2
Paused: 0
Stopped: 0
Images: 32
Server Version: 20.10.17
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: journald
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
Default Runtime: runc
Init Binary: /usr/libexec/docker/docker-init
containerd version:
runc version: e910b9f-dirty
init version:
Security Options:
seccomp
Profile: default
selinux
cgroupns
Kernel Version: 5.18.5-200.fc36.aarch64
Operating System: Fedora CoreOS 36.20220618.3.1
OSType: linux
Architecture: aarch64
CPUs: 4
Total Memory: 1.835GiB
Name: localhost.localdomain
ID: VUN3:MOT3:OYXF:Q4QC:BJIP:7CA5:ND5I:APJ6:6T2A:QMPI:E5TF:OWC3
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: true
-----> git version: git version 2.25.1
-----> sigil version: 0.9.0build+bc921b7
-----> herokuish version:
! herokuish not supported on arm64 architecture
-----> dokku version: dokku version 0.27.7
-----> plugn version: plugn: 0.12.0build+3a27594
-----> dokku plugins:
00_dokku-standard 0.27.7 enabled dokku core standard plugin
20_events 0.27.7 enabled dokku core events logging plugin
app-json 0.27.7 enabled dokku core app-json plugin
apps 0.27.7 enabled dokku core apps plugin
builder 0.27.7 enabled dokku core builder plugin
builder-dockerfile 0.27.7 enabled dokku core builder-dockerfile plugin
builder-herokuish 0.27.7 enabled dokku core builder-herokuish plugin
builder-null 0.27.7 enabled dokku core builder-null plugin
builder-pack 0.27.7 enabled dokku core builder-pack plugin
buildpacks 0.27.7 enabled dokku core buildpacks plugin
certs 0.27.7 enabled dokku core certificate management plugin
checks 0.27.7 enabled dokku core checks plugin
common 0.27.7 enabled dokku core common plugin
config 0.27.7 enabled dokku core config plugin
cron 0.27.7 enabled dokku core cron plugin
docker-options 0.27.7 enabled dokku core docker-options plugin
domains 0.27.7 enabled dokku core domains plugin
enter 0.27.7 enabled dokku core enter plugin
git 0.27.7 enabled dokku core git plugin
letsencrypt 0.16.3 enabled Automated installation of let's encrypt TLS certificates
logs 0.27.7 enabled dokku core logs plugin
network 0.27.7 enabled dokku core network plugin
nginx-vhosts 0.27.7 enabled dokku core nginx-vhosts plugin
plugin 0.27.7 enabled dokku core plugin plugin
postgres 1.22.0 enabled dokku postgres service plugin
proxy 0.27.7 enabled dokku core proxy plugin
ps 0.27.7 enabled dokku core ps plugin
registry 0.27.7 enabled dokku core registry plugin
repo 0.27.7 enabled dokku core repo plugin
resource 0.27.7 enabled dokku core resource plugin
run 0.27.7 enabled dokku core run plugin
scheduler 0.27.7 enabled dokku core scheduler plugin
scheduler-docker-local 0.27.7 enabled dokku core scheduler-docker-local plugin
scheduler-null 0.27.7 enabled dokku core scheduler-null plugin
shell 0.27.7 enabled dokku core shell plugin
ssh-keys 0.27.7 enabled dokku core ssh-keys plugin
storage 0.27.7 enabled dokku core storage plugin
trace 0.27.7 enabled dokku core trace plugin
=====> check app-json information
App json computed selected: app.json
App json global selected: app.json
App json selected:
=====> check app information
App created at: 1657571833
App deploy source: check
App deploy source metadata: check
App dir: /home/dokku/check
App locked: false
=====> check builder information
Builder build dir:
Builder computed build dir:
Builder computed selected:
Builder global build dir:
Builder global selected:
Builder selected:
=====> check builder-dockerfile information
Builder dockerfile computed dockerfile path: Dockerfile
Builder dockerfile global dockerfile path: Dockerfile
Builder dockerfile dockerfile path:
=====> check builder-pack information
Builder pack computed projecttoml path: project.toml
Builder pack global projecttoml path: project.toml
Builder pack projecttoml path:
=====> check buildpacks information
Buildpacks computed stack: gliderlabs/herokuish:latest-20
Buildpacks global stack:
Buildpacks list:
Buildpacks stack:
=====> check ssl information
Ssl dir: /home/dokku/check/tls
Ssl enabled: false
Ssl hostnames:
Ssl expires at:
Ssl issuer:
Ssl starts at:
Ssl subject:
Ssl verified:
=====> check checks information
Checks disabled list: none
Checks skipped list: none
=====> check cron information
Cron task count: 0
=====> check docker options information
Docker options build:
Docker options deploy: --restart=on-failure:10
Docker options run:
=====> check domains information
Domains app enabled: true
Domains app vhosts: check.silicon.am
Domains global enabled: true
Domains global vhosts: silicon.am
=====> check git information
Git deploy branch: main
Git global deploy branch: master
Git keep git dir: false
Git rev env var: GIT_REV
Git sha:
Git last updated at: 1657571835
Can't open /home/dokku/check/tls/server.crt for reading, No such file or directory
281473335814624:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:69:fopen('/home/dokku/check/tls/server.crt','r')
281473335814624:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:76:
unable to load certificate
=====> check letsencrypt information
Letsencrypt active: false
Letsencrypt autorenew: false
Letsencrypt email: domains@assembled.app
Letsencrypt expiration: 1657497600
=====> check logs information
Logs computed max size: 10m
Logs global max size: 10m
Logs global vector sink:
Logs max size:
Logs vector sink:
=====> check network information
Network attach post create:
Network attach post deploy:
Network bind all interfaces: false
Network computed attach post create:
Network computed attach post deploy:
Network computed bind all interfaces: false
Network computed initial network:
Network computed tld:
Network global attach post create:
Network global attach post deploy:
Network global bind all interfaces: false
Network global initial network:
Network global tld:
Network initial network:
Network static web listener:
Network tld:
Network web listeners: 172.17.0.3:80
=====> check nginx information
Nginx access log format:
Nginx access log path: /var/log/nginx/check-access.log
Nginx bind address ipv4:
Nginx bind address ipv6: ::
Nginx client max body size:
Nginx disable custom config: false
Nginx error log path: /var/log/nginx/check-error.log
Nginx global hsts: true
Nginx computed hsts: true
Nginx hsts:
Nginx hsts include subdomains: true
Nginx hsts max age: 15724800
Nginx hsts preload: false
Nginx proxy buffer size: 4096
Nginx proxy buffering: on
Nginx proxy buffers: 8 4096
Nginx proxy busy buffers size: 8192
Nginx proxy read timeout: 60s
Nginx last visited at: 1657570214
Nginx x forwarded for value: $remote_addr
Nginx x forwarded port value: $server_port
Nginx x forwarded proto value: $scheme
Nginx x forwarded ssl:
=====> check proxy information
Proxy enabled: true
Proxy port map: http:80:80
Proxy type: nginx
=====> check ps information
Deployed: true
Processes: 1
Ps can scale: true
Ps computed procfile path: Procfile
Ps global procfile path: Procfile
Ps procfile path:
Ps restart policy: on-failure:10
Restore: true
Running: true
Status web 1: running (CID: cc2e8f0f17f)
=====> check registry information
Registry computed image repo: dokku/check
Registry computed push on release: false
Registry computed server:
Registry global push on release:
Registry global server:
Registry image repo:
Registry push on release:
Registry server:
Registry tag version:
=====> check resource information
=====> check scheduler information
Scheduler computed selected: docker-local
Scheduler global selected: docker-local
Scheduler selected:
=====> check scheduler-docker-local information
Scheduler docker local disable chown:
Scheduler docker local parallel schedule count:
=====> check storage information
Storage build mounts:
Storage deploy mounts:
Storage run mounts: dokku ps:inspect $APP
[core@localhost ~]$ dokku ps:inspect check
[
{
"AppArmorProfile": "",
"Args": [
"nginx",
"-g",
"daemon off;"
],
"Config": {
"AttachStderr": true,
"AttachStdin": false,
"AttachStdout": true,
"Cmd": [
"nginx",
"-g",
"daemon off;"
],
"Domainname": "",
"Entrypoint": [
"/docker-entrypoint.sh"
],
"Env": [
"APPLICATION_HOST=XXXXXX",
"DOKKU_APP_TYPE=dockerfile",
"DOKKU_DOCKERFILE_PORTS=80",
"DOKKU_LETSENCRYPT_EMAIL=domains@assembled.app",
"GIT_REV=XXXXXX",
"DYNO=web.1",
"PORT=80",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"NGINX_VERSION=XXXXXX",
"NJS_VERSION=XXXXXX",
"PKG_RELEASE=XXXXXX"
],
"ExposedPorts": {
"80/tcp": {}
},
"Hostname": "cc2e8f0f17f2",
"Image": "dokku/check:latest",
"Labels": {
"com.dokku.app-name": "check",
"com.dokku.builder-type": "dockerfile",
"com.dokku.container-type": "deploy",
"com.dokku.dyno": "web.1",
"com.dokku.image-stage": "release",
"com.dokku.process-type": "web",
"dokku": "",
"maintainer": "NGINX Docker Maintainers <docker-maint@nginx.com>",
"org.label-schema.schema-version": "1.0",
"org.label-schema.vendor": "dokku"
},
"OnBuild": null,
"OpenStdin": false,
"StdinOnce": false,
"StopSignal": "SIGQUIT",
"Tty": false,
"User": "",
"Volumes": null,
"WorkingDir": ""
},
"Created": "2022-07-11T20:36:42.337375622Z",
"Driver": "overlay2",
"ExecIDs": null,
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/fcf2759bbddad21b24bc1e8b4c37424392210e3fe555b2dbcab6551ef3b88555-init/diff:/var/lib/docker/overlay2/3131a4af5dce4a7341f51241752e989ff15d740a420bc65bd6d71a974db6a920/diff:/var/lib/docker/overlay2/5674e2e4dd173a224acbda33a3efe424a5758cd3b79dab097f03df73ce7bbd27/diff:/var/lib/docker/overlay2/86121c9266dcf8d84c3f26246902dea464044571ea1da6cd85338a810333e201/diff:/var/lib/docker/overlay2/9c77dbd208c610cccbb5b031675e886c67a0d6168c920ec6240af1457d166dee/diff:/var/lib/docker/overlay2/15e5d22ff5b7da7203f9720452af96ce37bce6ca9179ce908eb5f0507ff9bd1d/diff:/var/lib/docker/overlay2/d86612ac024412efde45ed8d239b1c3e15ce7f41c99b857086d4cf5660c91c84/diff:/var/lib/docker/overlay2/cc961d4717b6458945618703da93d286e50376ae947e71dbd927e704f586e027/diff",
"MergedDir": "/var/lib/docker/overlay2/fcf2759bbddad21b24bc1e8b4c37424392210e3fe555b2dbcab6551ef3b88555/merged",
"UpperDir": "/var/lib/docker/overlay2/fcf2759bbddad21b24bc1e8b4c37424392210e3fe555b2dbcab6551ef3b88555/diff",
"WorkDir": "/var/lib/docker/overlay2/fcf2759bbddad21b24bc1e8b4c37424392210e3fe555b2dbcab6551ef3b88555/work"
},
"Name": "overlay2"
},
"HostConfig": {
"AutoRemove": false,
"Binds": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceWriteIOps": null,
"BlkioWeight": 0,
"BlkioWeightDevice": [],
"CapAdd": null,
"CapDrop": null,
"Cgroup": "",
"CgroupParent": "",
"CgroupnsMode": "private",
"ConsoleSize": [
0,
0
],
"ContainerIDFile": "",
"CpuCount": 0,
"CpuPercent": 0,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpuShares": 0,
"CpusetCpus": "",
"CpusetMems": "",
"DeviceCgroupRules": null,
"DeviceRequests": null,
"Devices": [],
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IOMaximumBandwidth": 0,
"IOMaximumIOps": 0,
"Init": true,
"IpcMode": "private",
"Isolation": "",
"KernelMemory": 0,
"KernelMemoryTCP": 0,
"Links": null,
"LogConfig": {
"Config": {},
"Type": "journald"
},
"MaskedPaths": [
"/proc/asound",
"/proc/acpi",
"/proc/kcore",
"/proc/keys",
"/proc/latency_stats",
"/proc/timer_list",
"/proc/timer_stats",
"/proc/sched_debug",
"/proc/scsi",
"/sys/firmware"
],
"Memory": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": null,
"NanoCpus": 0,
"NetworkMode": "default",
"OomKillDisable": null,
"OomScoreAdj": 0,
"PidMode": "",
"PidsLimit": null,
"PortBindings": {},
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyPaths": [
"/proc/bus",
"/proc/fs",
"/proc/irq",
"/proc/sys",
"/proc/sysrq-trigger"
],
"ReadonlyRootfs": false,
"RestartPolicy": {
"MaximumRetryCount": 10,
"Name": "on-failure"
},
"Runtime": "runc",
"SecurityOpt": null,
"ShmSize": 67108864,
"UTSMode": "",
"Ulimits": [
{
"Hard": 1024,
"Name": "nofile",
"Soft": 1024
}
],
"UsernsMode": "",
"VolumeDriver": "",
"VolumesFrom": null
},
"HostnamePath": "/var/lib/docker/containers/cc2e8f0f17f26af6ddcf93da167be388f8a5320233235bf1225e7d90eed785eb/hostname",
"HostsPath": "/var/lib/docker/containers/cc2e8f0f17f26af6ddcf93da167be388f8a5320233235bf1225e7d90eed785eb/hosts",
"Id": "cc2e8f0f17f26af6ddcf93da167be388f8a5320233235bf1225e7d90eed785eb",
"Image": "sha256:1530ef28fd6b6fd9398756ae38af5d45c18413462bad7da5e82087cbe7741d69",
"LogPath": "",
"MountLabel": "system_u:object_r:container_file_t:s0:c288,c529",
"Mounts": [],
"Name": "/check.web.1",
"NetworkSettings": {
"Bridge": "",
"EndpointID": "4031731dfbcb853ef88255590934ca3c3964945d3cfa0b57dfbf9335730774f0",
"Gateway": "172.17.0.1",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"HairpinMode": false,
"IPAddress": "172.17.0.3",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:11:00:03",
"Networks": {
"bridge": {
"Aliases": null,
"DriverOpts": null,
"EndpointID": "4031731dfbcb853ef88255590934ca3c3964945d3cfa0b57dfbf9335730774f0",
"Gateway": "172.17.0.1",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAMConfig": null,
"IPAddress": "172.17.0.3",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"Links": null,
"MacAddress": "02:42:ac:11:00:03",
"NetworkID": "f868feb8b624410045a75e4d48e9faeaa6819d81811d131a9818e5c2c537b1b9"
}
},
"Ports": {
"80/tcp": null
},
"SandboxID": "8104e345f3616f72dc3b513b11f9ff62a55a2302ad085908e3ee1525414ff09a",
"SandboxKey": "/var/run/docker/netns/8104e345f361",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null
},
"Path": "/docker-entrypoint.sh",
"Platform": "linux",
"ProcessLabel": "system_u:system_r:container_t:s0:c288,c529",
"ResolvConfPath": "/var/lib/docker/containers/cc2e8f0f17f26af6ddcf93da167be388f8a5320233235bf1225e7d90eed785eb/resolv.conf",
"RestartCount": 0,
"State": {
"Dead": false,
"Error": "",
"ExitCode": 0,
"FinishedAt": "0001-01-01T00:00:00Z",
"OOMKilled": false,
"Paused": false,
"Pid": 32510,
"Restarting": false,
"Running": true,
"StartedAt": "2022-07-11T20:36:44.03789472Z",
"Status": "running"
}
}
]
I fixed it by using older version 0.17.0
sudo dokku plugin:install https://github.com/dokku/dokku-letsencrypt.git --committish 0.17.0
@satalsky that isnt older than what he is using. Are you sure you're describing this issue and not something else?
Hello! I am having same issue. I tried to debug it, here is log with trace enabled. I tried to debug it, it seems like proxy is working just alright, but nothing listens from another side. It is possibly because dokku is ran inside docker, not on the host machine.
EDIT: Oh, it seems that this time it failed due to validation limit. Previous failure was because of 502 engine proxy failure as in first report:
acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: 140.238.137.166: Invalid response from http://hello-world.app.svokkkaaa.com/.well-known/acme-challenge/wy2z4yYQ39eg4NxNeCN-odJiD_7fEKbQtWoss0NvVqU: 502
I am currently experiencing the same issue, any solutions?