Timescaledb restarts multiple time just after startup of the docker
daniel1302 opened this issue · comments
system info
Linux Ubuntu 22.04
AMD64
docker version
Client: Docker Engine - Community
Cloud integration: v1.0.35-desktop+001
Version: 24.0.5
API version: 1.43
Go version: go1.20.6
Git commit: ced0996
Built: Fri Jul 21 20:35:45 2023
OS/Arch: linux/amd64
Context: default
Server: Docker Desktop
Engine:
Version: 24.0.5
API version: 1.43 (minimum version 1.12)
Go version: go1.20.6
Git commit: a61e2b4
Built: Fri Jul 21 20:35:45 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.21
GitCommit: 3dce8eb055cbb6872793272b4f20ed16117344f8
runc:
Version: 1.1.7
GitCommit: v1.1.7-0-g860f061
docker-init:
Version: 0.19.0
GitCommit: de40ad0
I have the following Dockerfile:
FROM timescale/timescaledb:2.8.1-pg14
ENV TIMESCALEDB_TELEMETRY="off"
ENV NO_TS_TUNE="true"
RUN set -euo; \
sed -i 's/^restart_after_crash.*//' /usr/local/share/postgresql/postgresql.conf.sample\
&& echo "restart_after_crash=no" >> /usr/local/share/postgresql/postgresql.conf.sample;
COPY 020_add_multiple_databases.sh /docker-entrypoint-initdb.d/My 020_add_multiple_databases.sh script looks like following:
!/bin/sh
IFS=","
for v in $POSTGRES_DBS
do
PGPASSWORD=${POSTGRES_PASSWORD} psql -U ${POSTGRES_USER} -c "CREATE DATABASE $v" -c "GRANT ALL PRIVILEGES ON DATABASE $v TO ${POSTGRES_USER}"
doneThen When I call the following:
docker build -t timescaledb:2.8.0-pg14-patch2 . && docker run -e POSTGRES_USER=postgres -e POSTGRES_DBS=db1,db2,db3,db4 -e POSTGRES_PASSWORD=somepassword timescaledb:2.8.0-pg14-patch2I am getting the following output:
[+] Building 0.6s (8/8) FINISHED docker:default
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 464B 0.0s
=> [internal] load metadata for docker.io/timescale/timescaledb:2.8.1-pg14 0.5s
=> [1/3] FROM docker.io/timescale/timescaledb:2.8.1-pg14@sha256:96c3a5587a24bb336437403b394f5927f639322cf3fcc0eff4ee0294da540dd7 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 51B 0.0s
=> CACHED [2/3] RUN set -euo; sed -i 's/^restart_after_crash.*//' /usr/local/share/postgresql/postgresql.conf.sample && 0.0s
=> CACHED [3/3] COPY 020_add_multiple_databases.sh /docker-entrypoint-initdb.d/ 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:9b10c3bf32f7594528cc1f9d5b8048dac426fbb1dd5f50b12eecf75254b249c2 0.0s
=> => naming to docker.io/library/timescaledb:2.8.0-pg14-patch2 0.0s
What's Next?
View summary of image vulnerabilities and recommendations → docker scout quickview
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... sh: locale: not found
2023-08-23 16:54:05.013 UTC [30] WARNING: no usable system locales were found
ok
syncing data to disk ... ok
Success. You can now start the database server using:
pg_ctl -D /var/lib/postgresql/data -l logfile start
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
waiting for server to start....2023-08-23 16:54:06.026 UTC [36] LOG: starting PostgreSQL 14.5 on x86_64-pc-linux-musl, compiled by gcc (Alpine 11.2.1_git20220219) 11.2.1 20220219, 64-bit
2023-08-23 16:54:06.031 UTC [36] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-08-23 16:54:06.044 UTC [37] LOG: database system was shut down at 2023-08-23 16:54:05 UTC
2023-08-23 16:54:06.050 UTC [36] LOG: database system is ready to accept connections
2023-08-23 16:54:06.051 UTC [43] LOG: TimescaleDB background worker launcher connected to shared catalogs
done
server started
/usr/local/bin/docker-entrypoint.sh: sourcing /docker-entrypoint-initdb.d/000_install_timescaledb.sh
psql:/tmp/tmp.fPGKkL:1: WARNING:
WELCOME TO
_____ _ _ ____________
|_ _(_) | | | _ \ ___ \
| | _ _ __ ___ ___ ___ ___ __ _| | ___| | | | |_/ /
| | | | _ ` _ \ / _ \/ __|/ __/ _` | |/ _ \ | | | ___ \
| | | | | | | | | __/\__ \ (_| (_| | | __/ |/ /| |_/ /
|_| |_|_| |_| |_|\___||___/\___\__,_|_|\___|___/ \____/
Running version 2.8.1
For more information on TimescaleDB, please visit the following links:
1. Getting started: https://docs.timescale.com/timescaledb/latest/getting-started
2. API reference documentation: https://docs.timescale.com/api/latest
3. How TimescaleDB is designed: https://docs.timescale.com/timescaledb/latest/overview/core-concepts
Note: TimescaleDB collects anonymous reports to better understand and assist our users.
For more information and how to disable, please see our docs https://docs.timescale.com/timescaledb/latest/how-to-guides/configuration/telemetry.
2023-08-23 16:54:06.472 UTC [53] WARNING:
WELCOME TO
_____ _ _ ____________
|_ _(_) | | | _ \ ___ \
| | _ _ __ ___ ___ ___ ___ __ _| | ___| | | | |_/ /
| | | | _ ` _ \ / _ \/ __|/ __/ _` | |/ _ \ | | | ___ \
| | | | | | | | | __/\__ \ (_| (_| | | __/ |/ /| |_/ /
|_| |_|_| |_| |_|\___||___/\___\__,_|_|\___|___/ \____/
Running version 2.8.1
For more information on TimescaleDB, please visit the following links:
1. Getting started: https://docs.timescale.com/timescaledb/latest/getting-started
2. API reference documentation: https://docs.timescale.com/api/latest
3. How TimescaleDB is designed: https://docs.timescale.com/timescaledb/latest/overview/core-concepts
Note: TimescaleDB collects anonymous reports to better understand and assist our users.
For more information and how to disable, please see our docs https://docs.timescale.com/timescaledb/latest/how-to-guides/configuration/telemetry.
2023-08-23 16:54:06.472 UTC [53] CONTEXT: PL/pgSQL function inline_code_block line 16 at RAISE
CREATE EXTENSION
alter_job
-------------------------------------------------
(1,24:00:00,00:01:40,-1,01:00:00,f,,-infinity,)
(1 row)
2023-08-23 16:54:06.837 UTC [56] WARNING:
WELCOME TO
_____ _ _ ____________
|_ _(_) | | | _ \ ___ \
| | _ _ __ ___ ___ ___ ___ __ _| | ___| | | | |_/ /
| | | | _ ` _ \ / _ \/ __|/ __/ _` | |/ _ \ | | | ___ \
| | | | | | | | | __/\__ \ (_| (_| | | __/ |/ /| |_/ /
|_| |_|_| |_| |_|\___||___/\___\__,_|_|\___|___/ \____/
Running version 2.8.1
For more information on TimescaleDB, please visit the following links:
1. Getting started: https://docs.timescale.com/timescaledb/latest/getting-started
2. API reference documentation: https://docs.timescale.com/api/latest
3. How TimescaleDB is designed: https://docs.timescale.com/timescaledb/latest/overview/core-concepts
Note: TimescaleDB collects anonymous reports to better understand and assist our users.
For more information and how to disable, please see our docs https://docs.timescale.com/timescaledb/latest/how-to-guides/configuration/telemetry.
2023-08-23 16:54:06.837 UTC [56] CONTEXT: PL/pgSQL function inline_code_block line 16 at RAISE
psql:/tmp/tmp.fPGKkL:1: WARNING:
WELCOME TO
_____ _ _ ____________
|_ _(_) | | | _ \ ___ \
| | _ _ __ ___ ___ ___ ___ __ _| | ___| | | | |_/ /
| | | | _ ` _ \ / _ \/ __|/ __/ _` | |/ _ \ | | | ___ \
| | | | | | | | | __/\__ \ (_| (_| | | __/ |/ /| |_/ /
|_| |_|_| |_| |_|\___||___/\___\__,_|_|\___|___/ \____/
Running version 2.8.1
For more information on TimescaleDB, please visit the following links:
1. Getting started: https://docs.timescale.com/timescaledb/latest/getting-started
2. API reference documentation: https://docs.timescale.com/api/latest
3. How TimescaleDB is designed: https://docs.timescale.com/timescaledb/latest/overview/core-concepts
Note: TimescaleDB collects anonymous reports to better understand and assist our users.
For more information and how to disable, please see our docs https://docs.timescale.com/timescaledb/latest/how-to-guides/configuration/telemetry.
2023-08-23 16:54:06.842 UTC [57] ERROR: TimescaleDB background worker connected to template database, exiting
CREATE EXTENSION
2023-08-23 16:54:06.842 UTC [36] LOG: background worker "TimescaleDB Background Worker Scheduler" (PID 57) exited with exit code 1
alter_job
-------------------------------------------------
(1,24:00:00,00:01:40,-1,01:00:00,f,,-infinity,)
(1 row)
/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/001_timescaledb_tune.sh
/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/020_add_multiple_databases.sh
CREATE DATABASE
GRANT
CREATE DATABASE
GRANT
CREATE DATABASE
GRANT
CREATE DATABASE
GRANT
waiting for server to shut down...2023-08-23 16:54:08.445 UTC [36] LOG: received fast shutdown request
.2023-08-23 16:54:08.450 UTC [36] LOG: aborting any active transactions
2023-08-23 16:54:08.450 UTC [54] FATAL: terminating background worker "TimescaleDB Background Worker Scheduler" due to administrator command
2023-08-23 16:54:08.450 UTC [43] FATAL: terminating background worker "TimescaleDB Background Worker Launcher" due to administrator command
2023-08-23 16:54:08.451 UTC [36] LOG: background worker "logical replication launcher" (PID 44) exited with exit code 1
2023-08-23 16:54:08.451 UTC [36] LOG: background worker "TimescaleDB Background Worker Launcher" (PID 43) exited with exit code 1
2023-08-23 16:54:08.451 UTC [36] LOG: background worker "TimescaleDB Background Worker Scheduler" (PID 54) exited with exit code 1
2023-08-23 16:54:08.451 UTC [38] LOG: shutting down
2023-08-23 16:54:08.500 UTC [36] LOG: database system is shut down
done
server stopped
PostgreSQL init process complete; ready for start up.
2023-08-23 16:54:08.567 UTC [1] LOG: starting PostgreSQL 14.5 on x86_64-pc-linux-musl, compiled by gcc (Alpine 11.2.1_git20220219) 11.2.1 20220219, 64-bit
2023-08-23 16:54:08.567 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2023-08-23 16:54:08.567 UTC [1] LOG: listening on IPv6 address "::", port 5432
2023-08-23 16:54:08.580 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-08-23 16:54:08.590 UTC [70] LOG: database system was shut down at 2023-08-23 16:54:08 UTC
2023-08-23 16:54:08.597 UTC [1] LOG: database system is ready to accept connections
2023-08-23 16:54:08.598 UTC [76] LOG: TimescaleDB background worker launcher connected to shared catalogsMy question is why TimescaleDB restarts so many times, and how I could debug it?
Some times it takes seconds and sometimes about 1 minute
It's critical part because I am starting database during the tests, so i want this step to be faster.
@daniel1302 Does this problem occur with the base image as well? Unfortunately, we cannot help with custom Dockerfiles and startup scripts. I suggest you try removing your changes one-by-one until it works. That's one way to identify the issue.
The issue will be closed. Feel free to reopen if you find a specific issue/bug in our Docker image.