Caddy service can't start
ashkadov86 opened this issue · comments
I'm have ubuntu 20.04. After last upgrade system service caddy can't start with state Exit 2.
i tested on multiple hosts with ubuntu 20.04 with latest updates, same problem everywhere.
On ububntu 18.04 there are no such problems.
What could be the reason?
I'm having the same issue.
+1
I am facing the same issue
log
goroutine 1 [running]:
github.com/mholt/caddy/vendor/github.com/miekg/dns.ClientConfigFromFile(0xbb4739, 0x10, 0x0, 0x0, 0x0)
src/github.com/mholt/caddy/vendor/github.com/miekg/dns/clientconfig.go:86 +0xad6
github.com/mholt/caddy/vendor/github.com/xenolf/lego/acme.getNameservers(0xbb4739, 0x10, 0xfeaf20, 0x2, 0x2, 0xf77460, 0xc4200402c0, 0xc420037f50)
src/github.com/mholt/caddy/vendor/github.com/xenolf/lego/acme/dns_challenge.go:40 +0x4d
github.com/mholt/caddy/vendor/github.com/xenolf/lego/acme.init()
src/github.com/mholt/caddy/vendor/github.com/xenolf/lego/acme/dns_challenge.go:33 +0x12d
github.com/mholt/caddy/caddy/caddymain.init()
:1 +0x75
main.init()
:1 +0x44
panic: runtime error: slice bounds out of range
I changed the network dockprom_monitor-net ownership from administrator to public and re-deploy caddy container
works for me
Can users with the issue please report the output of:
docker versiondocker info
With the information you have provided I would suspect it's one of the packages you upgraded that's causing the issue.
docker version
Client: Docker Engine - Community
Version: 20.10.3
API version: 1.41
Go version: go1.13.15
Git commit: 48d30b5
Built: Fri Jan 29 14:33:21 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.3
API version: 1.41 (minimum version 1.12)
Go version: go1.13.15
Git commit: 46229ca
Built: Fri Jan 29 14:31:32 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.3
GitCommit: 269548fa27e0089a8b8278fc4fc781d7f65a939b
runc:
Version: 1.0.0-rc92
GitCommit: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
docker-init:
Version: 0.19.0
GitCommit: de40ad0
docker info
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
Server:
Containers: 7
Running: 0
Paused: 0
Stopped: 7
Images: 13
Server Version: 20.10.3
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.4.0-26-generic
Operating System: Ubuntu 20.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.843GiB
Name: prod-big-break-monitoring
ID: 26GE:SEC5:MACD:SID6:DUF3:C5SD:3RNH:OZND:EDNA:XAMI:7DK6:NF5P
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
WARNING: No blkio weight support
WARNING: No blkio weight_device support
On centos work fine
Same.
Fresh Ubuntu 20.04 LTS
This is some kind of global problem:
https://stackoverflow.com/questions/66074495/i-am-trying-to-run-caddy-server-with-docker-on-ubuntu-20-04-2-lts-but-caddy-not
zuohuadong/docker-caddy#5
I'm have ubuntu 20.04. After last upgrade system service caddy can't start with state Exit 2.
i tested on multiple hosts with ubuntu 20.04 with latest updates, same problem everywhere.
On ububntu 18.04 there are no such problems.
What could be the reason?
My solution was to use the original caddy image
OK, i've figured this out @nightah
The problem is that as of ubuntu 20.04 'trust-ad' is added to the end of resolv.conf. Could others confirm that /etc/resolv.conf ends with?
options edns0 trust-ad
Mine is:
nameserver 127.0.0.53
options edns0 trust-ad
When docker runs it copies the resolv.conf from the host. However, the trust-ad throws an error in caddy because of this bug in the main go dns module they import. This was fixed here:
But caddy uses a version prior:
https://github.com/caddyserver/caddy/search?q=github.com%2Fmiekg%2Fdns
A quick fix it to somehow overwrite the resolv.conf that docker uses. Docker makes this slightly complicated, but the easiest way i found is to create a copy of /etc/resolv.conf somewhere, and edit it to remove the last trust-ad and change the nameserver ip to 127.0.0.11 (docker dns server default) - see below. Then change your docker-compose.yml and import it as a volume with read-only permissions where resolv.conf normally lives. You may also have to add ndots:0 - something docker appends after it copies the file for some complicated dns reason.
My edited resolv.conf looks like:
nameserver 127.0.0.11
options edns0 ndots:0
docker-compose.yml under services: caddy edit volumes to:
volumes:
- ./caddy:/etc/caddy
- /path/to/local/resolv.conf:/etc/resolv.conf:ro
Result: container starts as expected.
An alternative fix would be to manually edit your resolv.conf in place on the system, so the copy that is brought across works. I haven't tried to do that because of the way ubuntu manages this file that i don't completely understand.
This is fixed in version 2 of caddy
Thanks for the write up @DigiDr, this issue has been resolved with v3.30.0.