Container doesn't wait for its dependency container to be ready & healthy

[Terry Tu Nguyen]

I have 3 containers:

• my_mongo: Mongo DB
• my_mongo_express: a GUI web server of Mongo. This will connect to my_mongo
• my_mongo_checker: this is for debugging to see if my_mongo works & exposes its host & port correctly

The problem is: my_mongo_express started & crashed before my_mongo became ready

My tox.ini file:

# tox.ini

[tox]
envlist = py39,lint
skipsdist = true

[docker:my_mongo]
image = mongo:5.0
environment =
    MONGO_INITDB_ROOT_USERNAME=aaa
    MONGO_INITDB_ROOT_PASSWORD=bbb
    MONGO_INITDB_DATABASE=ccc
healthcheck_cmd = echo 'db.runCommand("ping").ok' | mongo mongodb://aaa:bbb@localhost:27017 --quiet
healthcheck_interval = 20
healthcheck_timeout = 60
healthcheck_retries = 30
healthcheck_start_period = 2

[docker:my_mongo_express]
image = mongo-express
environment =
    ME_CONFIG_MONGODB_SERVER=my_mongo
    ME_CONFIG_MONGODB_ADMINUSERNAME=aaa
    ME_CONFIG_MONGODB_ADMINPASSWORD=bbb
ports =
    8081:8081/tcp
links =
    my_mongo

[docker:my_mongo_checker]
image = mongo:5.0
environment =
    MONGO_INITDB_ROOT_USERNAME=qqq
    MONGO_INITDB_ROOT_PASSWORD=www
    MONGO_INITDB_DATABASE=eee
links =
    my_mongo

[testenv:integration-tests]
allowlist_externals=
    /bin/sleep
    /bin/echo
deps =
    -r{toxinidir}/requirements/test.txt
    -r{toxinidir}/src/api_endpoint/requirements.txt
    -r{toxinidir}/src/person_enrichment_service/requirements.txt
setenv =
    ENVIRONMENT=testing
passenv =
    SALESFORCE_APP_ID
    SALESFORCE_APP_SECRET
commands =
    python -m pip --version
    sleep 10000000
docker =
    my_mongo
    my_mongo_express
    my_mongo_checker

---

Debugging info:

my_mongo_express crashed:

> docker ps -a                            
CONTAINER ID   IMAGE          COMMAND                  CREATED          STATUS                             PORTS                      NAMES
9c93735bc374   e02771e6e39b   "docker-entrypoint.s…"   11 seconds ago   Up 9 seconds                       0.0.0.0:32783->27017/tcp   my_mongo_checker-tox-22580
5dc7d97dbb97   4475769c9d8b   "tini -- /docker-ent…"   11 seconds ago   Exited (0) 9 seconds ago                                      my_mongo_express-tox-22580
eb74901bb52d   e02771e6e39b   "docker-entrypoint.s…"   13 seconds ago   Up 11 seconds (health: starting)   0.0.0.0:32782->27017/tcp   my_mongo-tox-22580

my_mongo_express said it could not connect my_mongo:

> docker logs my_mongo_express-tox-22580

Welcome to mongo-express
------------------------


(node:7) [MONGODB DRIVER] Warning: Current Server Discovery and Monitoring engine is deprecated, and will be removed in a future version. To use the new Server Discover and Monitoring engine, pass option { useUnifiedTopology: true } to the MongoClient constructor.
Could not connect to database using connectionString: mongodb://aaa:bbb@my_mongo:27017/"
(node:7) UnhandledPromiseRejectionWarning: MongoNetworkError: failed to connect to server [my_mongo:27017] on first connect [Error: connect ECONNREFUSED 172.17.0.2:27017
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1144:16) {
  name: 'MongoNetworkError'
}]
    at Pool.<anonymous> (/node_modules/mongodb/lib/core/topologies/server.js:441:11)
    at Pool.emit (events.js:314:20)
    at /node_modules/mongodb/lib/core/connection/pool.js:564:14
    at /node_modules/mongodb/lib/core/connection/pool.js:1000:11
    at /node_modules/mongodb/lib/core/connection/connect.js:32:7
    at callback (/node_modules/mongodb/lib/core/connection/connect.js:300:5)
    at Socket.<anonymous> (/node_modules/mongodb/lib/core/connection/connect.js:330:7)
    at Object.onceWrapper (events.js:421:26)
    at Socket.emit (events.js:314:20)
    at emitErrorNT (internal/streams/destroy.js:92:8)
    at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)
    at processTicksAndRejections (internal/process/task_queues.js:84:21)
(node:7) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:7) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

my_mongo works correctly

docker exec -it my_mongo_checker-tox-22580 /bin/bash
root@9c93735bc374:/# mongo mongodb://aaa:bbb@my_mongo:27017^C
root@9c93735bc374:/# echo 'db.runCommand("ping").ok' | mongo mongodb://aaa:bbb@my_mongo:27017 --quiet
1

---

Version:

Tox: 4.5.1
Tox-docker: 4.1.0

/opt/homebrew/lib/python3.9/site-packages/requests/__init__.py:102: RequestsDependencyWarning: urllib3 (1.26.7) or chardet (5.1.0)/charset_normalizer (2.0.12) doesn't match a supported version!
  warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported "
4.5.1 from /opt/homebrew/lib/python3.9/site-packages/tox/__init__.py
registered plugins:
    tox-docker-4.1.0 at /opt/homebrew/lib/python3.9/site-packages/tox_docker/__init__.py

[Dan Crosta]

Docker doesn't have a notion of "dependencies" in the way you want here, and tox-docker doesn't either.

You could:

• change the my_mongo_express container's command to check & wait for the mongo DB to come up (this is somewhat easier in recent versions of tox-docker with the new dockerfile= directive)
• use docker-compose and commands_pre in your tox.ini. Unfortunately, docker-compose doesn't have a stable Python API (it's only a command-line tool), so tox-docker can't support it