Connection issues, when using docker-dind service at Gitlab CI

[Igor Davydenko]

Long story short

At Gitlab CI, when docker-dind service used, docker pull ... command works, while await docker.images.pull(...) does not.

• Expected behaviour: await docker.images.pull("playpauseandstop/docker-python:3.2.0-py38") works as well as docker pull -q playpauseandstop/docker-python:3.2.0-py38
• Actual behaviour: DockerError(404, "page not found")

How to reproduce

I have a project, which uses aiodocker and want to test it at Gitlab CI.

For achieving that, I'm trying to use docker-dind service with TLS enabled, as described in docs. However I'm unable to do requests to the docker from aiodocker, cause all them ends with DockerError(404, "page not found") error.

In same time docker ... commands executes well within same Gitlab CI job.

Test Job definition

image: "playpauseandstop/docker-python:3.2.0-py38"

test:
  stage: "test"
  cache:
    <<: *cache
    policy: "pull"
  variables:
    DOCKER_CERT_PATH: "/certs/client"
    DOCKER_DRIVER: "overlay2"
    DOCKER_HOST: "tcp://docker:2376/"
    DOCKER_TLS_CERTDIR: "/certs"
    DOCKER_TLS_VERIFY: "1"
    PYTHONPATH: "."
  services:
    - "docker:19.03.8-dind"
  before_script:
    - "apt update -y"
    - "apt install -y apt-transport-https ca-certificates curl gnupg-agent software-properties-common"
    - "curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -"
    - 'add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable"'
    - "apt update -y"
    - "apt install -y docker-ce-cli"
    - "docker info"
  script:
    - "docker pull -q playpauseandstop/docker-python:3.2.0-py38"
    - "${MAKE} test-only"
  after_script:
    - "docker images"

It makes me wonder, why script below works (test.script[0]),

docker pull -q playpauseandstop/docker-python:3.2.0-py38

When next code does not (inside of test.script[1]),

async def main() -> int:
    image = "playpauseandstop/docker-python:3.2.0-py38"

    try:
        async with Docker() as docker:  # type: ignore
            await docker.images.pull(image)
    except DockerError:
        logger.error(f"Unable to pull docker image {image!r}", exc_info=True)
        return 1

    logger.info(f"Docker image is ready: {image!r}")
    logger.info("All OK!")

    return 0

Full output of test.script,

$ docker pull -q playpauseandstop/docker-python:3.2.0-py38
docker.io/playpauseandstop/docker-python:3.2.0-py38
$ ${MAKE} test-only
DOCKER_IMAGE=playpauseandstop/docker-python:3.2.0-py38 poetry run python ./scripts/prepare-image.py
2020-05-17 13:33:13,133 [ERROR:scripts.prepare-image] Unable to pull docker image 'playpauseandstop/docker-python:3.2.0-py38'
Traceback (most recent call last):
  File "./scripts/prepare-image.py", line 19, in main
    await docker.images.pull(image)
  File "/builds/group/project/.venv/lib/python3.8/site-packages/aiodocker/images.py", line 133, in _handle_list
    async with cm as response:
  File "/builds/group/project/.venv/lib/python3.8/site-packages/aiodocker/utils.py", line 309, in __aenter__
    resp = await self._coro
  File "/builds/group/project/.venv/lib/python3.8/site-packages/aiodocker/docker.py", line 264, in _do_query
    raise DockerError(response.status, json.loads(what.decode("utf8")))
aiodocker.exceptions.DockerError: DockerError(404, 'page not found')
make: *** [Makefile:40: prepare-image-only] Error 1

I'm reusing same env vars between test.script[0] & test.script[1],

• DOCKER_HOST
• DOCKER_TLS_VERIFY
• DOCKER_CERT_PATH

And not sure why it results in "page not found" error. Any ideas on what I'm missing?

Thanks in advance.

ps. At localhost, everything works fine.

Your environment

• Gitlab Runner: 12.10.2
• Docker Service: docker:19.03.8-dind
• Python Version: 3.8.2

[Andrew Svetlov]

Sorry, I have no idea what's wrong with dind