Unable to update Netbox to v3.6 - container unhealthy
patrysma opened this issue · comments
Current Behavior
When I try to upgrade my local Netbox instance to the latest version: v.3.6, I receive error:
[root@netboxsandbox netbox-docker]# docker-compose up -d
Creating network "netbox-docker_default" with the default driver
Creating netbox-docker_redis-cache_1 ... done
Creating netbox-docker_redis_1 ... done
Creating netbox-docker_postgres_1 ... done
Creating netbox-docker_netbox_1 ... done
ERROR: for netbox-housekeeping Container "a02d8c7ab5c6" is unhealthy.
ERROR: for netbox-worker Container "a02d8c7ab5c6" is unhealthy.
After running docker ps:
[root@netboxsandbox netbox-docker]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a02d8c7ab5c6 netboxcommunity/netbox:v3.6-2.7.0 "/usr/bin/tini -- /o…" 9 minutes ago Up 9 minutes (unhealthy) 0.0.0.0:8000->8080/tcp netbox-docker_netbox_1
0fedf7afd081 postgres:15-alpine "docker-entrypoint.s…" 9 minutes ago Up 9 minutes 5432/tcp netbox-docker_postgres_1
97703fa6b03b redis:7-alpine "docker-entrypoint.s…" 9 minutes ago Up 9 minutes 6379/tcp netbox-docker_redis-cache_1
81d1962eb711 redis:7-alpine "docker-entrypoint.s…" 9 minutes ago Up 9 minutes 6379/tcp netbox-docker_redis_1
Container is up but it's unhealthy and application is not working, when I upgrade with database that has a lot of data.
Upgrade works, when database is empty.
Expected Behavior
I expected that upgrade will work with database full of data
Docker Compose Version
docker-compose version 1.28.5, build c4eb3a1f docker-py version: 4.4.4 CPython version: 3.7.10 OpenSSL version: OpenSSL 1.1.0l 10 Sep 2019
Docker Version
[root@netboxsandbox netbox-docker]# docker version
Client: Docker Engine - Community
Version: 20.10.5
API version: 1.41
Go version: go1.13.15
Git commit: 55c4c88
Built: Tue Mar 2 20:33:55 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.5
API version: 1.41 (minimum version 1.12)
Go version: go1.13.15
Git commit: 363e9a8
Built: Tue Mar 2 20:32:17 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.4
GitCommit: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
runc:
Version: 1.0.0-rc93
GitCommit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
docker-init:
Version: 0.19.0
GitCommit: de40ad0
The git Revision
The git Status
# On branch release
# Your branch is ahead of 'origin/release' by 451 commits.
# (use "git push" to publish your local commits)
#
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
#
# .docker-compose.override.yml.swp
nothing added to commit but untracked files present (use "git add" to track)
Startup Command
docker-compose up -d
NetBox Logs
Attaching to netbox-docker_netbox_1
netbox_1 | ⚙️ Applying database migrations
netbox_1 | 溺 loaded config '/etc/netbox/config/configuration.py'
netbox_1 | 溺 loaded config '/etc/netbox/config/extra.py'
netbox_1 | 溺 loaded config '/etc/netbox/config/logging.py'
netbox_1 | 溺 loaded config '/etc/netbox/config/plugins.py'
netbox_1 | Operations to perform:
netbox_1 | Apply all migrations: account, admin, auth, circuits, contenttypes, core, dcim, django_rq, extras, ipam, sessions, social_django, taggit, tenancy, users, virtualization, wireless
netbox_1 | Running migrations:
Content of docker-compose.override.yml
version: '3.4'
services:
netbox:
ports:
- 8000:8080
environment:
ALLOWED_HOSTS: "netbox-sandbox.domain.net localhost"
NETBOX_DELETE_LEGACY_DATA: 1
healthcheck:
start_period: 360s
Same here with version 3.6.0.
Migration process stuck.
For me this is happening when updating with a db full of data. No more migration after dcim.0175.
Fresh installation are smooth as always.
Maybe releated to netbox-community/netbox#13745 ?
Running migrations:
Applying users.0004_netboxgroup_netboxuser... OK
Applying account.0001_initial... OK
Applying dcim.0173_remove_napalm_fields... OK
Applying dcim.0174_device_latitude_device_longitude... OK
Applying dcim.0174_rack_starting_unit... OK
Applying dcim.0175_device_oob_ip... OK
Same here, it looks that with large database update is not possible, because migration process stuck
Running the migrations takes time when the database is large and/or on a slow machine. Please checkout the start_period settings and it's explanation:
https://github.com/netbox-community/netbox-docker/blob/release/docker-compose.override.yml.example
Why you guys always close issue and treat people as idiots without digging further or ask for more information?
Start period has nothing to do with this issue. Unhealthy is just the cause of netbox container get stucked during migration.
Bah....
Running the migrations takes time when the database is large and/or on a slow machine. Please checkout the
start_periodsettings and it's explanation: https://github.com/netbox-community/netbox-docker/blob/release/docker-compose.override.yml.example
I increased start_period to 1h, and it didn't help. Like @rizlas said container get stucked during migration..
I increased
start_periodto 1h, and it didn't help. Like @rizlas said container get stucked during migration..
Thanks for testing that out. Could you configure logging, to dig more into where exactly it get's stuck?
There was an issue in Netbox, where the migrations where inefficient with device types (netbox-community/netbox#13745). Maybe you're seeing this issue?
Running the migrations takes time when the database is large and/or on a slow machine. Please checkout the
start_periodsettings and it's explanation:
https://github.com/netbox-community/netbox-docker/blob/release/docker-compose.override.yml.example
Also, if you had taken a little more time to read instead of compulsively clicking "close issue", you would have read that I reported a related potential issue. As a developer my self, I would have made sure to try a reported error before answering, but I understand that we are not all the same.