Containers fails to start running on GHA
nefilim opened this issue · comments
I have tried filing an issue at testcontainers-java but they won't look at it since we use testcontainers-scala.
I thought it only potentially happens when running concurrently but we have one branch failing consistently now, not running concurrently with other actions.
Ok I see what's going on here - when looking at the failed container log output, it's only receiving STDERR output, for some reason STDOUT has not been connected?
Seems pretty odd for the container to log the message that's being waited for twice in the LogMessageWaitStrategy once on STDERR and once on STDOUT?
snippet from a working run:
STDERR: WARNING: enabling "trust" authentication for local connections
STDERR: You can change this by editing pg_hba.conf or using the option -A, or
STDERR: --auth-local and --auth-host, the next time you run initdb.
STDOUT: waiting for server to start....2022-10-12 21:56:44.641 UTC [48] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
STDOUT: 2022-10-12 21:56:44.648 UTC [49] LOG: database system was shut down at 2022-10-12 21:56:44 UTC
STDOUT: 2022-10-12 21:56:44.651 UTC [48] LOG: database system is ready to accept connections
STDOUT: done
STDOUT: server started
STDOUT: CREATE DATABASE
STDOUT:
STDOUT:
STDOUT: /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
STDOUT:
STDOUT: waiting for server to shut down...2022-10-12 21:56:45.020 UTC [48] LOG: received fast shutdown request
STDOUT: .2022-10-12 21:56:45.020 UTC [48] LOG: aborting any active transactions
STDOUT: 2022-10-12 21:56:45.023 UTC [48] LOG: background worker "logical replication launcher" (PID 55) exited with exit code 1
STDOUT: 2022-10-12 21:56:45.023 UTC [50] LOG: shutting down
STDOUT: 2022-10-12 21:56:45.032 UTC [48] LOG: database system is shut down
STDOUT: done
STDOUT: server stopped
STDOUT:
STDOUT: PostgreSQL init process complete; ready for start up.
STDOUT:
STDERR: 2022-10-12 21:56:45.133 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
STDERR: 2022-10-12 21:56:45.133 UTC [1] LOG: listening on IPv6 address "::", port 5432
STDERR: 2022-10-12 21:56:45.133 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
STDERR: 2022-10-12 21:56:45.141 UTC [76] LOG: database system was shut down at 2022-10-12 21:56:45 UTC
STDERR: 2022-10-12 21:56:45.143 UTC [1] LOG: database system is ready to accept connections
vs a broken run:
20:56:09.486 [io-compute-12] ERROR 🐳 [postgres:11] - Log output from the failed container:
WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
2022-10-12 20:55:10.614 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2022-10-12 20:55:10.614 UTC [1] LOG: listening on IPv6 address "::", port 5432
2022-10-12 20:55:10.614 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-10-12 20:55:10.624 UTC [77] LOG: database system was shut down at 2022-10-12 20:55:10 UTC
2022-10-12 20:55:10.627 UTC [1] LOG: database system is ready to accept connections
Hi @nefilim sorry to hear that you have problems with this. I'm afraid that I agree with you that it's not the scala wrapper problem. I don't see anything that overrides the core (java) lib behavior.
@kiview any thoughts? especially with the new information I found? Feels a little bit like a race condition...
@nefilim Thanks, I'll re-open the issue at testcontainers-java with the new information.
Regarding logging on STDOUT and STDERR for PotgreSQL, that's unfortunately how their image works. It is in parts documented in the discussion around this PR for more flexible PostgreSQL Wait Strategy:
testcontainers/testcontainers-java#5501 (comment)