fail tox run if health check fails
dcrosta opened this issue · comments
Currently, a failing health check skips running tests, but reports the tox run as passed ("commands succeeded" in green text in the terminal):
$ git diff
diff --git a/tox.ini b/tox.ini
index 8d33c01..e2b9354 100644
--- a/tox.ini
+++ b/tox.ini
@@ -41,7 +41,7 @@ deps = pytest
commands = py.test [] test_healthcheck.py
[docker:redis:5-alpine]
-healthcheck_cmd = redis-cli ping | grep -q PONG
+healthcheck_cmd = false
healthcheck_interval = 1
healthcheck_timeout = 1
healthcheck_retries = 30
$ tox -e healthcheck-custom
GLOB sdist-make: /Users/dcrosta/src/tox-docker/setup.py
healthcheck-custom inst-nodeps: /Users/dcrosta/src/tox-docker/.tox/.tmp/package/1/tox-docker-1.5.0.zip
healthcheck-custom installed: appdirs==1.4.3,attrs==19.3.0,certifi==2020.4.5.1,chardet==3.0.4,distlib==0.3.0,docker==4.2.0,filelock==3.0.12,idna==2.9,more-itertools==8.2.0,packaging==20.3,pluggy==0.13.1,py==1.8.1,pyparsing==2.4.7,pytest==5.4.1,requests==2.23.0,six==1.14.0,toml==0.10.0,tox==3.14.6,tox-docker==1.5.0,urllib3==1.25.9,virtualenv==20.0.18,wcwidth==0.1.9,websocket-client==0.57.0
healthcheck-custom docker: run 'redis:5-alpine'
healthcheck-custom docker: health check: 'redis:5-alpine'
____________________________________________________________________________________ summary _____________________________________________________________________________________
healthcheck-custom: commands succeeded
congratulations :)
Traceback (most recent call last):
File "/Users/dcrosta/.pyenv/versions/tox-docker-3.8/bin/tox", line 8, in <module>
sys.exit(cmdline())
[...]
File "/Users/dcrosta/src/tox-docker/tox_docker.py", line 218, in tox_runtest_pre
raise HealthCheckFailed("{!r} failed health check".format(image))
tox_docker.HealthCheckFailed: 'redis:5-alpine' failed health check
The existing code raises an exception, but Tox 3 seems to print-but-swallow that exception. Tox2 fails when the exception is raised, as was (originally) expected)
I noticed this while fixing #57 - do you have an idea of what is required to solve? Happy to put it together but it isn't immediately obvious to me how to fix
It seems like the test runner isn't picking up status codes? Does the script need to exit with tox return code?
Here is an example where pytest fails but the ci job succeeds that I just noticed:
2020-06-02T17:01:26.9243885Z ============================== 1 passed in 0.17s ===============================
2020-06-02T17:01:26.9585199Z ports docker: remove '6bb14c924f' (forced)
2020-06-02T17:01:27.2421897Z links inst-nodeps: /home/vsts/work/1/s/.tox/dist/tox-docker-1.6.0.post31.zip
2020-06-02T17:01:29.1765509Z links installed: appdirs==1.4.4,attrs==19.3.0,certifi==2020.4.5.1,chardet==3.0.4,distlib==0.3.0,docker==4.2.1,filelock==3.0.12,idna==2.9,more-itertools==8.3.0,packaging==20.4,pluggy==0.13.1,py==1.8.1,pyparsing==2.4.7,pytest==5.4.3,requests==2.23.0,six==1.15.0,toml==0.10.1,tox==3.15.1,tox-docker @ file:///home/vsts/work/1/s/.tox/dist/tox-docker-1.6.0.post31.zip,urllib3==1.25.9,virtualenv==20.0.21,wcwidth==0.2.3,websocket-client==0.57.0
2020-06-02T17:01:29.1970934Z links docker: run 'httpd:alpine'
2020-06-02T17:01:29.5535167Z links docker: run 'registry:2'
2020-06-02T17:01:29.8955794Z links docker: run 'nginx:alpine'
2020-06-02T17:01:30.3603373Z links runtests: PYTHONHASHSEED='870840864'
2020-06-02T17:01:30.3604083Z links runtests: commands[0] | py.test test_links.py
2020-06-02T17:01:30.5651891Z ============================= test session starts ==============================
2020-06-02T17:01:30.5653388Z platform linux -- Python 3.8.3, pytest-5.4.3, py-1.8.1, pluggy-0.13.1
2020-06-02T17:01:30.5653918Z rootdir: /home/vsts/work/1/s
2020-06-02T17:01:30.7676696Z collected 3 items
2020-06-02T17:01:30.7681155Z
2020-06-02T17:01:31.1175960Z test_links.py F.F [100%]
2020-06-02T17:01:31.1176279Z
2020-06-02T17:01:31.1179147Z =================================== FAILURES ===================================
2020-06-02T17:01:31.1181761Z ____________________ ToxDockerLinksTest.test_links_created _____________________
2020-06-02T17:01:31.1182028Z
2020-06-02T17:01:31.1184357Z self = <test_links.ToxDockerLinksTest testMethod=test_links_created>
2020-06-02T17:01:31.1184579Z
2020-06-02T17:01:31.1187107Z def test_links_created(self):
2020-06-02T17:01:31.1187405Z client = docker.from_env(version="auto")
2020-06-02T17:01:31.1189344Z httpd_container = None
2020-06-02T17:01:31.1189596Z registry_container = None
2020-06-02T17:01:31.1191054Z nginx_container = None
2020-06-02T17:01:31.1192836Z for container in client.containers.list():
2020-06-02T17:01:31.1196293Z if httpd_container is None and "httpd" in container.attrs['Config']['Image']:
2020-06-02T17:01:31.1196709Z httpd_container = container
2020-06-02T17:01:31.1198802Z
2020-06-02T17:01:31.1200082Z if registry_container is None and "registry" in container.attrs['Config']['Image']:
2020-06-02T17:01:31.1201643Z registry_container = container
2020-06-02T17:01:31.1203724Z
2020-06-02T17:01:31.1205413Z if nginx_container is None and "nginx" in container.attrs['Config']['Image']:
2020-06-02T17:01:31.1215530Z nginx_container = container
2020-06-02T17:01:31.1216616Z
2020-06-02T17:01:31.1222645Z if all([httpd_container, registry_container, nginx_container]):
2020-06-02T17:01:31.1223377Z break
2020-06-02T17:01:31.1224049Z
2020-06-02T17:01:31.1229567Z self.assertIsNotNone(httpd_container, "could not find httpd container")
2020-06-02T17:01:31.1230142Z self.assertIsNotNone(registry_container, "could not find registry container")
2020-06-02T17:01:31.1230628Z self.assertIsNotNone(nginx_container, "could not find nginx container")
2020-06-02T17:01:31.1231922Z
2020-06-02T17:01:31.1234870Z httpd_name = httpd_container.attrs["Name"]
2020-06-02T17:01:31.1235445Z registry_name = registry_container.attrs["Name"]
2020-06-02T17:01:31.1242759Z nginx_name = nginx_container.attrs["Name"]
2020-06-02T17:01:31.1243253Z
2020-06-02T17:01:31.1244961Z httpd_links = httpd_container.attrs["HostConfig"]["Links"]
2020-06-02T17:01:31.1247747Z registry_links = registry_container.attrs["HostConfig"]["Links"]
2020-06-02T17:01:31.1248637Z nginx_links = nginx_container.attrs["HostConfig"]["Links"]
2020-06-02T17:01:31.1251098Z
2020-06-02T17:01:31.1251908Z self.assertIsNone(httpd_links)
2020-06-02T17:01:31.1254203Z
2020-06-02T17:01:31.1254741Z expected_registry_links = [
2020-06-02T17:01:31.1259781Z "{}:{}/{}".format(httpd_name, registry_name, 'httpd')
2020-06-02T17:01:31.1260321Z ]
2020-06-02T17:01:31.1263513Z > self.assertEqual(expected_registry_links, registry_links)
2020-06-02T17:01:31.1264616Z E AssertionError: Lists differ: ['/ecstatic_goldstine:/nostalgic_ramanujan/httpd'] != ['/ecstatic_goldstine:/nostalgic_ramanujan/apache']
2020-06-02T17:01:31.1265542Z E
2020-06-02T17:01:31.1265909Z E First differing element 0:
2020-06-02T17:01:31.1266742Z E '/ecstatic_goldstine:/nostalgic_ramanujan/httpd'
2020-06-02T17:01:31.1267443Z E '/ecstatic_goldstine:/nostalgic_ramanujan/apache'
2020-06-02T17:01:31.1267818Z E
2020-06-02T17:01:31.1268521Z E - ['/ecstatic_goldstine:/nostalgic_ramanujan/httpd']
2020-06-02T17:01:31.1269338Z E ? ^^^^
2020-06-02T17:01:31.1269832Z E
2020-06-02T17:01:31.1270379Z E + ['/ecstatic_goldstine:/nostalgic_ramanujan/apache']
2020-06-02T17:01:31.1270850Z E ? ++++ ^
2020-06-02T17:01:31.1271160Z
2020-06-02T17:01:31.1271440Z test_links.py:45: AssertionError
2020-06-02T17:01:31.1275124Z __________ ToxDockerLinksTest.test_validate_link_line_requires_alias ___________
2020-06-02T17:01:31.1275558Z
2020-06-02T17:01:31.1275945Z self = <test_links.ToxDockerLinksTest testMethod=test_validate_link_line_requires_alias>
2020-06-02T17:01:31.1276323Z
2020-06-02T17:01:31.1276674Z def test_validate_link_line_requires_alias(self):
2020-06-02T17:01:31.1277079Z for line in (
2020-06-02T17:01:31.1277852Z 'some-image-name',
2020-06-02T17:01:31.1278458Z 'some-image-name:',
2020-06-02T17:01:31.1278804Z ):
2020-06-02T17:01:31.1279156Z with self.assertRaises(ValueError) as cm:
2020-06-02T17:01:31.1279719Z > _validate_link_line(line)
2020-06-02T17:01:31.1280102Z E AssertionError: ValueError not raised
2020-06-02T17:01:31.1280387Z
2020-06-02T17:01:31.1280653Z test_links.py:80: AssertionError
2020-06-02T17:01:31.1670383Z =========================== short test summary info ============================
2020-06-02T17:01:31.1673189Z FAILED test_links.py::ToxDockerLinksTest::test_links_created - AssertionError...
2020-06-02T17:01:31.1673887Z FAILED test_links.py::ToxDockerLinksTest::test_validate_link_line_requires_alias
2020-06-02T17:01:31.1674419Z ========================= 2 failed, 1 passed in 0.60s ==========================
2020-06-02T17:01:31.2186786Z ERROR: InvocationError: '/home/vsts/work/1/s/.tox/links/bin/py.test test_links.py'
2020-06-02T17:01:31.2187715Z links docker: remove '060a00cff3' (forced)
2020-06-02T17:01:31.4188711Z links docker: remove '7d199004ad' (forced)
2020-06-02T17:01:31.6271688Z links docker: remove 'd8301eaac2' (forced)
2020-06-02T17:01:31.8622122Z ___________________________________ summary ____________________________________
2020-06-02T17:01:31.8622514Z integration: commands succeeded
2020-06-02T17:01:31.8622741Z registry: commands succeeded
2020-06-02T17:01:31.8623627Z healthcheck-builtin: commands succeeded
2020-06-02T17:01:31.8624076Z healthcheck-custom: commands succeeded
2020-06-02T17:01:31.8624313Z ports: commands succeeded
2020-06-02T17:01:31.8624923Z ERROR: links: commands failed
2020-06-02T17:01:31.9015160Z
2020-06-02T17:01:31.9087481Z ##[section]Finishing: Tox 2.x, Docker 4.x
Yeah, I ran out of time this morning to figure out the details here, but it's close...