App can't connect to db after dokku upgrade from 0.22.1 to 0.26.6
Hasenn opened this issue · comments
Description of problem
One of our applications can no longer connect to its postgres database after updating dokku from 0.22.1 to 0.26.6
we've deployed both the app and its pg service again and the app still can't. We've rebuilt everything after the upgrade, and restarted the pg services after updating the plugin
Environment Information
dokku 0.26.6
postgres plugin is updated to 1.17.0
dokku report APP_NAME output
~# dokku report polyauth2
-----> uname: Linux srv-dev.do.intra 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
-----> memory:
total used free shared buff/cache available
Mem: 63921 2622 50538 101 10760 60560
Swap: 8191 0 8191
-----> docker version:
Client: Docker Engine - Community
Version: 20.10.11
API version: 1.41
Go version: go1.16.9
Git commit: dea9396
Built: Thu Nov 18 00:37:06 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.11
API version: 1.41 (minimum version 1.12)
Go version: go1.16.9
Git commit: 847da18
Built: Thu Nov 18 00:35:15 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.12
GitCommit: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
runc:
Version: 1.0.2
GitCommit: v1.0.2-0-g52b36a2
docker-init:
Version: 0.19.0
GitCommit: de40ad0
-----> docker daemon info:
Client:
Context: default
Debug Mode: true
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.6.3-docker)
scan: Docker Scan (Docker Inc., v0.9.0)
Server:
Containers: 48
Running: 25
Paused: 0
Stopped: 23
Images: 300
Server Version: 20.10.11
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
runc version: v1.0.2-0-g52b36a2
WARNING: No swap limit support
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.4.0-90-generic
Operating System: Ubuntu 20.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 62.42GiB
Name: srv-dev.do.intra
ID: MHS5:CGKM:EVRX:DGNP:K62I:FA7P:UOTJ:P3NW:YDZP:LJUA:L3VH:CIB3
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
-----> git version: git version 2.25.1
-----> sigil version: 0.7.1build+1ec3006
-----> herokuish version:
herokuish: 0.5.31
buildpacks:
heroku-buildpack-multi v1.0.0
heroku-buildpack-ruby v230
heroku-buildpack-nodejs v189
heroku-buildpack-clojure v87
heroku-buildpack-python v201
heroku-buildpack-java v69
heroku-buildpack-gradle v35
heroku-buildpack-scala v91
heroku-buildpack-play v26
heroku-buildpack-php v199
heroku-buildpack-go v156
buildpack-nginx v14
buildpack-null v3
-----> dokku version: dokku version 0.26.6
-----> plugn version: plugn: 0.7.1build+9bf216b
-----> dokku plugins:
00_dokku-standard 0.26.6 enabled dokku core standard plugin
20_events 0.26.6 enabled dokku core events logging plugin
acl 1.5.1 enabled dokku plugin that can be used to restrict push privileges for app to certain users
app-json 0.26.6 enabled dokku core app-json plugin
apps 0.26.6 enabled dokku core apps plugin
builder 0.26.6 enabled dokku core builder plugin
builder-dockerfile 0.26.6 enabled dokku core builder-dockerfile plugin
builder-herokuish 0.26.6 enabled dokku core builder-herokuish plugin
builder-null 0.26.6 enabled dokku core builder-null plugin
builder-pack 0.26.6 enabled dokku core builder-pack plugin
buildpacks 0.26.6 enabled dokku core buildpacks plugin
certs 0.26.6 enabled dokku core certificate management plugin
checks 0.26.6 enabled dokku core checks plugin
common 0.26.6 enabled dokku core common plugin
config 0.26.6 enabled dokku core config plugin
cron 0.26.6 enabled dokku core cron plugin
docker-options 0.26.6 enabled dokku core docker-options plugin
domains 0.26.6 enabled dokku core domains plugin
enter 0.26.6 enabled dokku core enter plugin
git 0.26.6 enabled dokku core git plugin
letsencrypt 0.12.1 enabled Automated installation of let's encrypt TLS certificates
logs 0.26.6 enabled dokku core logs plugin
maintenance 0.5.1 enabled Maintenance mode for apps
mongo 1.16.0 enabled dokku mongo service plugin
network 0.26.6 enabled dokku core network plugin
nginx-vhosts 0.26.6 enabled dokku core nginx-vhosts plugin
plugin 0.26.6 enabled dokku core plugin plugin
postgres 1.17.0 enabled dokku postgres service plugin
proxy 0.26.6 enabled dokku core proxy plugin
ps 0.26.6 enabled dokku core ps plugin
registry 0.26.6 enabled dokku core registry plugin
repo 0.26.6 enabled dokku core repo plugin
resource 0.26.6 enabled dokku core resource plugin
run 0.26.6 enabled dokku core run plugin
scheduler 0.25.6 enabled dokku core scheduler plugin
scheduler-docker-local 0.26.6 enabled dokku core scheduler-docker-local plugin
scheduler-null 0.26.6 enabled dokku core scheduler-null plugin
shell 0.26.6 enabled dokku core shell plugin
ssh-keys 0.26.6 enabled dokku core ssh-keys plugin
storage 0.26.6 enabled dokku core storage plugin
trace 0.26.6 enabled dokku core trace plugin
=====> polyauth2 app-json information
App json computed selected: app.json
App json global selected: app.json
App json selected:
=====> polyauth2 app information
App created at: 1637771617
App deploy source: polyauth2
App deploy source metadata: polyauth2
App dir: /home/dokku/polyauth2
App locked: false
=====> polyauth2 builder information
Builder build dir:
Builder computed build dir:
Builder computed selected:
Builder global build dir:
Builder global selected:
Builder selected:
=====> polyauth2 builder-dockerfile information
Builder dockerfile computed dockerfile path: Dockerfile
Builder dockerfile global dockerfile path: Dockerfile
Builder dockerfile dockerfile path:
=====> polyauth2 builder-pack information
Builder pack computed projecttoml path: project.toml
Builder pack global projecttoml path: project.toml
Builder pack projecttoml path:
=====> polyauth2 buildpacks information
Buildpacks computed stack: gliderlabs/herokuish:latest-20
Buildpacks global stack:
Buildpacks list:
Buildpacks stack:
=====> polyauth2 ssl information
Ssl dir: /home/dokku/polyauth2/tls
Ssl enabled: true
Ssl hostnames: polyauth2.dev.dopolytech.fr
Ssl expires at: Feb 22 09:32:12 2022 GMT
Ssl issuer: C = US, O = Let's Encrypt, CN = R3
Ssl starts at: Nov 24 09:32:13 2021 GMT
Ssl subject: subject=CN = polyauth2.dev.dopolytech.fr
Ssl verified: self signed
=====> polyauth2 checks information
Checks disabled list: none
Checks skipped list: none
=====> polyauth2 cron information
Cron task count: 0
=====> polyauth2 docker options information
Docker options build: --link dokku.postgres.polyauth2-db:dokku-postgres-polyauth2-db
Docker options deploy: --link dokku.postgres.polyauth2-db:dokku-postgres-polyauth2-db --restart=on-failure:10
Docker options run: --link dokku.postgres.polyauth2-db:dokku-postgres-polyauth2-db
=====> polyauth2 domains information
Domains app enabled: true
Domains app vhosts: polyauth2.dev.dopolytech.fr
Domains global enabled: true
Domains global vhosts: dev.dopolytech.fr
=====> polyauth2 git information
Git deploy branch: master
Git global deploy branch: master
Git keep git dir: false
Git rev env var: GIT_REV
Git sha: 5950469
Git last updated at: 1637771617
=====> polyauth2 logs information
Logs computed max size: 10m
Logs global max size: 10m
Logs global vector sink:
Logs max size:
Logs vector sink:
=====> polyauth2 network information
Network attach post create:
Network attach post deploy:
Network bind all interfaces: false
Network computed attach post create:
Network computed attach post deploy:
Network computed bind all interfaces: false
Network computed initial network:
Network computed tld:
Network global attach post create:
Network global attach post deploy:
Network global bind all interfaces: false
Network global initial network:
Network global tld:
Network initial network:
Network static web listener:
Network tld:
Network web listeners: 172.17.0.21:5000
=====> polyauth2 nginx information
Nginx access log format:
Nginx access log path: /var/log/nginx/polyauth2-access.log
Nginx bind address ipv4:
Nginx bind address ipv6: ::
Nginx client max body size:
Nginx disable custom config: false
Nginx error log path: /var/log/nginx/polyauth2-error.log
Nginx global hsts: true
Nginx computed hsts: true
Nginx hsts:
Nginx hsts include subdomains: true
Nginx hsts max age: 15724800
Nginx hsts preload: false
Nginx proxy buffer size: 4096
Nginx proxy buffering: on
Nginx proxy buffers: 8 4096
Nginx proxy busy buffers size: 8192
Nginx proxy read timeout: 60s
Nginx last visited at: 1637772813
Nginx x forwarded for value: $remote_addr
Nginx x forwarded port value: $server_port
Nginx x forwarded proto value: $scheme
Nginx x forwarded ssl:
=====> polyauth2 proxy information
Proxy enabled: true
Proxy port map: http:80:5000 https:443:5000
Proxy type: nginx
=====> polyauth2 ps information
Deployed: true
Processes: 1
Ps can scale: true
Ps computed procfile path: Procfile
Ps global procfile path: Procfile
Ps procfile path:
Ps restart policy: on-failure:10
Restore: true
Running: true
Status web 1: running (CID: 0c6f71f58aa)
=====> polyauth2 registry information
Registry computed image repo: dokku/polyauth2
Registry computed push on release: false
Registry computed server:
Registry global push on release:
Registry global server:
Registry image repo:
Registry push on release:
Registry server:
Registry tag version:
=====> polyauth2 resource information
=====> polyauth2 scheduler information
Scheduler computed selected: docker-local
Scheduler global selected: docker-local
Scheduler selected:
=====> polyauth2 scheduler-docker-local information
Scheduler docker local disable chown:
Scheduler docker local parallel schedule count:
=====> polyauth2 storage information
Storage build mounts:
Storage deploy mounts:
Storage run mounts: How (deb/make/rpm) and where (AWS, VirtualBox, physical, etc.) was Dokku installed?:
using dokku's bootstrap.sh on a physical node
Additional information
- App container inspect output (if applicable) via
dokku ps:inspect APP_NAME
~# dokku ps:inspect polyauth2
[
{
"AppArmorProfile": "docker-default",
"Args": [
"web"
],
"Config": {
"AttachStderr": true,
"AttachStdin": false,
"AttachStdout": true,
"Cmd": [
"/start",
"web"
],
"Domainname": "",
"Entrypoint": null,
"Env": [
"DOKKU_APP_RESTORE=1",
"DOKKU_APP_TYPE=herokuish",
"LDAP_URL=XXXXXX",
"DYNO=web.1",
"CURL_TIMEOUT=XXXXXX",
"CURL_CONNECT_TIMEOUT=XXXXXX",
"DOKKU_LETSENCRYPT_EMAIL=admin@redacted",
"GIT_REV=XXXXXX",
"LOGIN_URL=XXXXXX",
"PORT=5000",
"USER=herokuishuser",
"DOKKU_PROXY_SSL_PORT=443",
"LDAP_PORT=XXXXXX",
"DATABASE_URL=XXXXXX",
"DOKKU_PROXY_PORT_MAP=http:80:5000 https:443:5000",
"DOKKU_DOCKERFILE_PORTS=8080",
"CACHE_PATH=/cache",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"STACK=XXXXXX",
"DEBIAN_FRONTEND=XXXXXX"
],
"Hostname": "0c6f71f58aa7",
"Image": "dokku/polyauth2:latest",
"Labels": {
"com.dokku.app-name": "polyauth2",
"com.dokku.container-type": "deploy",
"com.dokku.dyno": "web.1",
"com.dokku.image-stage": "release",
"com.dokku.process-type": "web",
"com.gliderlabs.herokuish/stack": "heroku-20",
"dokku": "",
"org.label-schema.schema-version": "1.0",
"org.label-schema.vendor": "dokku"
},
"OnBuild": null,
"OpenStdin": false,
"StdinOnce": false,
"Tty": false,
"User": "",
"Volumes": null,
"WorkingDir": ""
},
"Created": "2021-11-24T16:32:42.078699778Z",
"Driver": "overlay2",
"ExecIDs": null,
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/7d765e6890da3df6f2e3e3ef8f1e39803d74cdf9781199743144dc6aa336c4e9-init/diff:/var/lib/docker/overlay2/0993cbc07a57ec183ff67213679a45b4a1777505c0acae6030640cc47c57ed00/diff:/var/lib/docker/overlay2/e43a8719003e86260315409bdfb628bd9aba3bc7167661a0717a7d18102bc8ef/diff:/var/lib/docker/overlay2/65383c7634d71ca394a6f94e91b75200342f307d2063283521da5996511c63c7/diff:/var/lib/docker/overlay2/1e3b548d6e1b00cbe4cbb710f9647a0b75e943fbc9f3c9a9adc7efcb034234cb/diff:/var/lib/docker/overlay2/02075ad9599a4a07d75b485d68b5016dd447eba603962699efbd019f9433da49/diff:/var/lib/docker/overlay2/71fa32e4db2c00ec962e0cb7bbc249d0e76136f7ba117e4c562dd26e60a715ee/diff:/var/lib/docker/overlay2/d5b2b4e844d229ba45616874e00abfa6eeb3cf946964687d422c1a529ec1f93d/diff:/var/lib/docker/overlay2/eb78dc71a24d24401c3f3af38794862d0ecc1440fb4fb5de0beddf49fd3752a4/diff:/var/lib/docker/overlay2/b8df3c8596832b19a4b4949fe4589a8b0cb39f5453c23931187021ff330d8cc2/diff:/var/lib/docker/overlay2/fa4f9c51523ab615da7016c98f88a0c9c3b1650af93417c45b9b4df30c3b3fe2/diff:/var/lib/docker/overlay2/6adfdfb0ecfba358f16e21e36cc871bf9b91d75aa9affb170542d53be7fc39e7/diff:/var/lib/docker/overlay2/aad63bc90a715b1ef5a4721aa73faae505582cccf0da962dc5419cdf95e49044/diff:/var/lib/docker/overlay2/c550024763445ef45a0e8c8086de487bc95fbfd901dc4de4a7208901d4841a4f/diff:/var/lib/docker/overlay2/4955fab201094bf9974efbf08275a3155686aa5bb227fc801175f1f271867a05/diff:/var/lib/docker/overlay2/072833762aa9a92d1874a71c5b09c5b757990cbfbe98b9a92727933ee8675689/diff",
"MergedDir": "/var/lib/docker/overlay2/7d765e6890da3df6f2e3e3ef8f1e39803d74cdf9781199743144dc6aa336c4e9/merged",
"UpperDir": "/var/lib/docker/overlay2/7d765e6890da3df6f2e3e3ef8f1e39803d74cdf9781199743144dc6aa336c4e9/diff",
"WorkDir": "/var/lib/docker/overlay2/7d765e6890da3df6f2e3e3ef8f1e39803d74cdf9781199743144dc6aa336c4e9/work"
},
"Name": "overlay2"
},
"HostConfig": {
"AutoRemove": false,
"Binds": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceWriteIOps": null,
"BlkioWeight": 0,
"BlkioWeightDevice": [],
"CapAdd": null,
"CapDrop": null,
"Cgroup": "",
"CgroupParent": "",
"CgroupnsMode": "host",
"ConsoleSize": [
0,
0
],
"ContainerIDFile": "",
"CpuCount": 0,
"CpuPercent": 0,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpuShares": 0,
"CpusetCpus": "",
"CpusetMems": "",
"DeviceCgroupRules": null,
"DeviceRequests": null,
"Devices": [],
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IOMaximumBandwidth": 0,
"IOMaximumIOps": 0,
"Init": true,
"IpcMode": "private",
"Isolation": "",
"KernelMemory": 0,
"KernelMemoryTCP": 0,
"Links": [
"/dokku.postgres.polyauth2-db:/polyauth2.web.1/dokku-postgres-polyauth2-db"
],
"LogConfig": {
"Config": {
"max-size": "10m"
},
"Type": "json-file"
},
"MaskedPaths": [
"/proc/asound",
"/proc/acpi",
"/proc/kcore",
"/proc/keys",
"/proc/latency_stats",
"/proc/timer_list",
"/proc/timer_stats",
"/proc/sched_debug",
"/proc/scsi",
"/sys/firmware"
],
"Memory": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": null,
"NanoCpus": 0,
"NetworkMode": "default",
"OomKillDisable": false,
"OomScoreAdj": 0,
"PidMode": "",
"PidsLimit": null,
"PortBindings": {},
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyPaths": [
"/proc/bus",
"/proc/fs",
"/proc/irq",
"/proc/sys",
"/proc/sysrq-trigger"
],
"ReadonlyRootfs": false,
"RestartPolicy": {
"MaximumRetryCount": 10,
"Name": "on-failure"
},
"Runtime": "runc",
"SecurityOpt": null,
"ShmSize": 67108864,
"UTSMode": "",
"Ulimits": null,
"UsernsMode": "",
"VolumeDriver": "",
"VolumesFrom": null
},
"HostnamePath": "/var/lib/docker/containers/0c6f71f58aa71f83c1664d58892a7097c65516a5c9ea790fb789e2ed4c0f9a1e/hostname",
"HostsPath": "/var/lib/docker/containers/0c6f71f58aa71f83c1664d58892a7097c65516a5c9ea790fb789e2ed4c0f9a1e/hosts",
"Id": "0c6f71f58aa71f83c1664d58892a7097c65516a5c9ea790fb789e2ed4c0f9a1e",
"Image": "sha256:5f14458562cabf4296351cd54b0dbd1eac7568035374f0bffcf2466cc219e9f9",
"LogPath": "/var/lib/docker/containers/0c6f71f58aa71f83c1664d58892a7097c65516a5c9ea790fb789e2ed4c0f9a1e/0c6f71f58aa71f83c1664d58892a7097c65516a5c9ea790fb789e2ed4c0f9a1e-json.log",
"MountLabel": "",
"Mounts": [],
"Name": "/polyauth2.web.1",
"NetworkSettings": {
"Bridge": "",
"EndpointID": "f0c5fbb3bf8d11b8a601abe1c05aec20e2a971d2978ad1f393b0aa124526b44b",
"Gateway": "172.17.0.1",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"HairpinMode": false,
"IPAddress": "172.17.0.21",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:11:00:15",
"Networks": {
"bridge": {
"Aliases": null,
"DriverOpts": null,
"EndpointID": "f0c5fbb3bf8d11b8a601abe1c05aec20e2a971d2978ad1f393b0aa124526b44b",
"Gateway": "172.17.0.1",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAMConfig": null,
"IPAddress": "172.17.0.21",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"Links": null,
"MacAddress": "02:42:ac:11:00:15",
"NetworkID": "b3adf07830afd55673e350137391ea7563aa9957a7f37afca5b6a5cd7e16550f"
}
},
"Ports": {},
"SandboxID": "f8eec4e9c3cdc400e4dd5ad318e5e3ef41a760e310a20144f06c10f070feb2d5",
"SandboxKey": "/var/run/docker/netns/f8eec4e9c3cd",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null
},
"Path": "/start",
"Platform": "linux",
"ProcessLabel": "",
"ResolvConfPath": "/var/lib/docker/containers/0c6f71f58aa71f83c1664d58892a7097c65516a5c9ea790fb789e2ed4c0f9a1e/resolv.conf",
"RestartCount": 0,
"State": {
"Dead": false,
"Error": "",
"ExitCode": 0,
"FinishedAt": "0001-01-01T00:00:00Z",
"OOMKilled": false,
"Paused": false,
"Pid": 2433352,
"Restarting": false,
"Running": true,
"StartedAt": "2021-11-24T16:32:44.418187034Z",
"Status": "running"
}
}
]
- The nginx configuration (if applicable) via
dokku nginx:show-config APP_NAME
~# dokku nginx:show-config polyauth2
server {
listen [::]:80;
listen 80;
server_name polyauth2.dev.dopolytech.fr;
access_log /var/log/nginx/polyauth2-access.log;
error_log /var/log/nginx/polyauth2-error.log;
include /home/dokku/polyauth2/nginx.conf.d/*.conf;
location / {
return 301 https://$host:443$request_uri;
}
}
server {
listen [::]:443 ssl http2;
listen 443 ssl http2;
server_name polyauth2.dev.dopolytech.fr;
access_log /var/log/nginx/polyauth2-access.log;
error_log /var/log/nginx/polyauth2-error.log;
ssl_certificate /home/dokku/polyauth2/tls/server.crt;
ssl_certificate_key /home/dokku/polyauth2/tls/server.key;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers off;
keepalive_timeout 70;
location / {
gzip on;
gzip_min_length 1100;
gzip_buffers 4 32k;
gzip_types text/css text/javascript text/xml text/plain text/x-component application/javascript application/x-javascript application/json application/xml application/rss+xml font/truetype application/x-font-ttf font/opentype application/vnd.ms-fontobject image/svg+xml;
gzip_vary on;
gzip_comp_level 6;
proxy_pass http://polyauth2-5000;
http2_push_preload on;
proxy_http_version 1.1;
proxy_read_timeout 60s;
proxy_buffer_size 4096;
proxy_buffering on;
proxy_buffers 8 4096;
proxy_busy_buffers_size 8192;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $http_connection;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Request-Start $msec;
}
include /home/dokku/polyauth2/nginx.conf.d/*.conf;
error_page 400 401 402 403 405 406 407 408 409 410 411 412 413 414 415 416 417 418 420 422 423 424 426 428 429 431 444 449 450 451 /400-error.html;
location /400-error.html {
root /var/lib/dokku/data/nginx-vhosts/dokku-errors;
internal;
}
error_page 404 /404-error.html;
location /404-error.html {
root /var/lib/dokku/data/nginx-vhosts/dokku-errors;
internal;
}
error_page 500 501 503 504 505 506 507 508 509 510 511 /500-error.html;
location /500-error.html {
root /var/lib/dokku/data/nginx-vhosts/dokku-errors;
internal;
}
error_page 502 /502-error.html;
location /502-error.html {
root /var/lib/dokku/data/nginx-vhosts/dokku-errors;
internal;
}
}
upstream polyauth2-5000 {
server 172.17.0.21:5000;
}
- Link to the exact repository being deployed (if possible/applicable):
private & hosted on our gitlab with internal auth - Output of failing Dokku commands after running
dokku trace:on
no command failed, just the app throwing errors and not connecting to db
Is the linked database running? Does it show any errors in the logs?
yes, it is running, and i don't see any errors when i do dokku postgres:logs polyauth2-db
Whats the log output from your database, and what are the errors you are seeing with your app?
PostgreSQL Database directory appears to contain a database; Skipping initialization
2021-11-22 15:44:18.932 UTC [1] LOG: starting PostgreSQL 13.4 (Debian 13.4-4.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2021-11-22 15:44:18.959 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2021-11-22 15:44:18.959 UTC [1] LOG: listening on IPv6 address "::", port 5432
2021-11-22 15:44:19.295 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-11-22 15:44:19.689 UTC [26] LOG: database system was shut down at 2021-11-22 15:38:40 UTC
2021-11-22 15:44:20.175 UTC [1] LOG: database system is ready to accept connectionsThe errors in my app are ugly, but they say that some isConnected thing is undefined (which happens on requests that do database things), and on startup it doesn't say it connected to the db like it should
If you run dokku postgres:connect $SERVICE, does it connect you to psql?
Also, can you print out the connection string the app is using from within code, and compare that to what is specified in the env vars for the app, and also compare to the the dsn url in the output of dokku postgres:info $SERVICE?
Yes it does connect to psql,
the code, config:show and the dsn url in dokku postgres:info polyauth2 are all the same including the db name
Can you show the entire log output of your app? If the config s all the same and the :connect command works, this feels like an app bug.
2021-11-25T15:59:07.750505978Z app[web.1]: [info] Auth server started in dev
2021-11-25T15:59:07.778882545Z app[web.1]: postgres://postgres:[REDACTED]@dokku-postgres-polyauth2-db:5432/polyauth2_db
2021-11-25T15:59:07.975891790Z app[web.1]: [info] Listening on port 5000
2021-11-25T15:59:07.989993767Z app[web.1]: [info] Connected to LDAP
2021-11-25T16:14:51.359977160Z app[web.1]: [error] read ECONNRESET # this is where it throws an error while connecting to the db
[...]
# When the request happens
2021-11-27T19:44:04.394219787Z app[web.1]: [debug] GET /authorize?client_id=admitech&state=85&redirect_uri=https%3A%2F%2F[REDACTED]%2Foauth - - ms - -
not sure the logs will help much
I'll look more closely into the code to see if i can find an app bug, It does work locally and that same code did work on the server before we upgraded dokku so its a bit puzzling.
Thanks a lot for your time !
i guess i'll also try going into the container and connecting to the db from there with the dburl
Okay, so the herokuish builder chooses what version of npm it wants, probably taking the latest stable release, and when it ran again after the upgrade it chose npm >= 14, which isn't compactible with pg-npm < 8 because of a bug that makes connecting to the db hang forever. Upgrading pg to >= 8 fixed it, and so nothing was wrong with either dokku or dokku-postgres
Closing this issue