Inegration tests failing in CI

[Taylor Silva]

Since April 3rd the integration test suite has been failing in the main CI pipeline, blocking all PR's and us doing another release. Will be investigating and posting results in this issue.

Logs from: https://ci.concourse-ci.org/teams/main/pipelines/concourse/jobs/integration/builds/753

=== EXEC docker-compose up -d --build

Recreating testpipelinepauser_db_1 ... 

[19565] Failed to execute script docker-compose


ERROR: for testpipelinepauser_db_1  'ContainerConfig'


ERROR: for db  'ContainerConfig'

Traceback (most recent call last):

  File "bin/docker-compose", line 3, in <module>

  File "compose/cli/main.py", line 67, in main

  File "compose/cli/main.py", line 126, in perform_command

  File "compose/cli/main.py", line 1070, in up

  File "compose/cli/main.py", line 1066, in up

  File "compose/project.py", line 648, in up

  File "compose/parallel.py", line 108, in parallel_execute

  File "compose/parallel.py", line 206, in producer

  File "compose/project.py", line 634, in do

  File "compose/service.py", line 579, in execute_convergence_plan

  File "compose/service.py", line 501, in _execute_convergence_recreate

  File "compose/parallel.py", line 108, in parallel_execute

  File "compose/parallel.py", line 206, in producer

  File "compose/service.py", line 494, in recreate

  File "compose/service.py", line 613, in recreate_container

  File "compose/service.py", line 332, in create_container

  File "compose/service.py", line 917, in _get_container_create_options

  File "compose/service.py", line 957, in _build_container_volume_options

  File "compose/service.py", line 1532, in merge_volume_bindings

  File "compose/service.py", line 1562, in get_container_data_volumes

KeyError: 'ContainerConfig'

==== exit status 255

==== EXEC docker-compose logs --no-color

==== EXEC docker-compose kill

Killing testpipelinepauser_worker_1 ... done

Killing testpipelinepauser_web_1    ... done

==== EXEC docker-compose down -v

Removing testpipelinepauser_worker_1          ... done

Removing testpipelinepauser_web_1             ... done

Removing 905931bf7144_testpipelinepauser_db_1 ... done

Removing network testpipelinepauser_default

Removing volume testpipelinepauser_postgresData

--- FAIL: TestPipelinePauser (303.06s)

    cmd.go:84: ExitCode 255 != 0

FAIL

FAIL	github.com/concourse/concourse/integration/pauser	303.155s

ok  	github.com/concourse/concourse/integration/worker	457.159s

FAIL

[Taylor Silva]

Checked to see if this changed started after installing a newer version of docker.

Before April 3rd the unit image had docker 25.0.3 installed: https://ci.concourse-ci.org/teams/main/pipelines/concourse/jobs/unit-image/builds/124
On April 3rd a new unit image was made and installed docker 26.0.0.

I currently have docker v26 installed on my machine. I haven't tried running the integration tests locally. Going to do that next and continue investigating locally.

[Taylor Silva]

We are installing a veryyy old version of docker-compose: https://github.com/concourse/ci/blob/883ba8faf5db4ae8af582143cd7d54f0a59e958b/dockerfiles/unit/Dockerfile#L47-L49
Guessing the drift has finally hit the breaking point here.

I think it may be worth trying to upgrade the integration test suite to use the built-in docker compose command. We already have it installed on the unit image because it comes with docker-ce.