Transient issues with (skopeo) push (fatal/EOF)

[phlax]

Problem description

Since around 12th April we have been seeing intermittent issues trying to push images to the registry

The issue varies and is intermittent (rerunning usually fixes) but at this point its failing about 80% of our CI

We use skopeo to push so the issue may be related to that, but we didnt change version recently so the change that causes the issue/s appears to be on the dockerhub side

Debug Information

URL:

CI pipeline with regular failure is here:

https://dev.azure.com/cncf/envoy/_build?definitionId=11

Our (envoy) ticket is here:

envoyproxy/envoy#33536

Timetamp or time range:

Public IP:

Hub Username:

phlax/envoyproxy/envoydockerbot

Error messages (on screen or in browser console)

Some examples:

time="2024-04-12T23:45:19Z" level=fatal msg="trying to reuse blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 at destination: Head \"[https://registry-1.docker.io/v2/envoyproxy/envoy-contrib-dev/blobs/sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1\"](https://registry-1.docker.io/v2/envoyproxy/envoy-contrib-dev/blobs/sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1/%22): EOF"

https://dev.azure.com/cncf/envoy/_build/results?buildId=167866&view=logs&j=86165412-e6cf-5df7-4573-670f16ea7b21&t=17984700-8a8b-562b-3e24-93462ef57afd&l=518

time="2024-04-16T19:43:03Z" level=fatal msg="writing manifest \"{\\n  \\\"schemaVersion\\\": 2,\\n  \\\"mediaType\\\": \\\"application/vnd.oci.image.manifest.v1+json\\\",\\n  \\\"config\\\": {\\n    \\\"mediaType\\\": \\\"application/vnd.oci.image.config.v1+json\\\",\\n    \\\"digest\\\": \\\"sha256:b0141e37894a8f255d0ac0060d58ae47d6bc1ce0c3a224970f5c8f4eceadb970\\\",\\n    \\\"size\\\": 2691\\n  },\\n  \\\"layers\\\": [\\n    {\\n      \\\"mediaType\\\": \\\"application/vnd.oci.image.layer.v1.tar+gzip\\\",\\n      \\\"digest\\\": \\\"sha256:d12dd1747fc756df543a17efb415b43b1aaac8074fe01b91c3d32c76946b6a49\\\",\\n      \\\"size\\\": 104180\\n    },\\n    {\\n      \\\"mediaType\\\": \\\"application/vnd.oci.image.layer.v1.tar+gzip\\\",\\n      \\\"digest\\\": \\\"sha256:e8d9a567199d7a318c875f2558a679ba8a924f817afacbb428afc3ffe6be6828\\\",\\n      \\\"size\\\": 13370\\n    },\\n    {\\n      \\\"mediaType\\\": \\\"application/vnd.oci.image.layer.v1.tar+gzip\\\",\\n      \\\"digest\\\": \\\"sha256:058cf3d8c2ba04ad7c064698c08c5e886a8623c0ad6171b8d72684253534417d\\\",\\n      \\\"size\\\": 537709\\n    },\\n    {\\n      \\\"mediaType\\\": \\\"application/vnd.oci.image.layer.v1.tar+gzip\\\",\\n      \\\"digest\\\": \\\"sha256:b6824ed73363f94b3b2b44084c51c31bc32af77a96861d49e16f91e3ab6bed71\\\",\\n      \\\"size\\\": 67\\n    },\\n    {\\n      \\\"mediaType\\\": \\\"application/vnd.oci.image.layer.v1.tar+gzip\\\",\\n      \\\"digest\\\": \\\"sha256:7c12895b777bcaa8ccae0605b4de635b68fc32d60fa08f421dc3818bf55ee212\\\",\\n      \\\"size\\\": 188\\n    },\\n    {\\n      \\\"mediaType\\\": \\\"application/vnd.oci.image.layer.v1.tar+gzip\\\",\\n      \\\"digest\\\": \\\"sha256:33e068de264953dfdc9f9ada207e76b61159721fd64a4820b320d05133a55fb8\\\",\\n      \\\"size\\\": 122\\n    },\\n    {\\n      \\\"mediaType\\\": \\\"application/vnd.oci.image.layer.v1.tar+gzip\\\",\\n      \\\"digest\\\": \\\"sha256:5664b15f108bf9436ce3312090a767300800edbbfd4511aa1a6d64357024d5dd\\\",\\n      \\\"size\\\": 168\\n    },\\n    {\\n      \\\"mediaType\\\": \\\"application/vnd.oci.image.layer.v1.tar+gzip\\\",\\n      \\\"digest\\\": \\\"sha256:27be814a09ebd97fac6fb7b82d19f117185e90601009df3fbab6f442f85cd6b3\\\",\\n      \\\"size\\\": 93\\n    },\\n    {\\n      \\\"mediaType\\\": \\\"application/vnd.oci.image.layer.v1.tar+gzip\\\",\\n      \\\"digest\\\": \\\"sha256:4aa0ea1413d37a58615488592a0b827ea4b2e48fa5a77cf707d0e35f025e613f\\\",\\n      \\\"size\\\": 385\\n    },\\n    {\\n      \\\"mediaType\\\": \\\"application/vnd.oci.image.layer.v1.tar+gzip\\\",\\n      \\\"digest\\\": \\\"sha256:da7816fa955ea24533c388143c78804c28682eef99b4ee3723b548c70148bba6\\\",\\n      \\\"size\\\": 321\\n    },\\n    {\\n      \\\"mediaType\\\": \\\"application/vnd.oci.image.layer.v1.tar+gzip\\\",\\n      \\\"digest\\\": \\\"sha256:9aee425378d2c16cd44177dc54a274b312897f5860a8e78fdfda555a0d79dd71\\\",\\n      \\\"size\\\": 130495\\n    },\\n    {\\n      \\\"mediaType\\\": \\\"application/vnd.oci.image.layer.v1.tar+gzip\\\",\\n      \\\"digest\\\": \\\"sha256:b2b365485a74c5f150939b44192fe85fe978dc32ef5bcaaa25be2a6d08099360\\\",\\n      \\\"size\\\": 5529318\\n    },\\n    {\\n      \\\"mediaType\\\": \\\"application/vnd.oci.image.layer.v1.tar+gzip\\\",\\n      \\\"digest\\\": \\\"sha256:34b8952c1d9dfa7d3a4b1bd7a65dbc65c9e3b3ddc437dd0a8a55351b1d600ada\\\",\\n      \\\"size\\\": 786\\n    },\\n    {\\n      \\\"mediaType\\\": \\\"application/vnd.oci.image.layer.v1.tar+gzip\\\",\\n      \\\"digest\\\": \\\"sha256:b7569544dea97ebdf27e2a54482f4d9d9f01b9921d6abed13d38bf79da1845bd\\\",\\n      \\\"size\\\": 24273849\\n    }\\n  ]\\n}\": Put \"[https://registry-1.docker.io/v2/envoyproxy/envoy-distroless-dev/manifests/sha256:5037d4109ed8ea9039b3ab2503ae44ba6ddf22389c548fe850c2c3c89b7924e7\"](https://registry-1.docker.io/v2/envoyproxy/envoy-distroless-dev/manifests/sha256:5037d4109ed8ea9039b3ab2503ae44ba6ddf22389c548fe850c2c3c89b7924e7/%22): EOF"

https://dev.azure.com/cncf/envoy/_build/results?buildId=168120&view=logs&j=86165412-e6cf-5df7-4573-670f16ea7b21&t=17984700-8a8b-562b-3e24-93462ef57afd&l=861

Screenshots of the issue (if applicable)

Task List

• This is NOT a security issue
• I do NOT have a Docker subscription
• I have looked through other issues and they do NOT apply to me

[phlax]

this seems to have gone away, ill close once we have a bit more confirmation