[Question] failed to send blob post - unauthorized

[Athos85]

Question

Hello everyone

I am trying a minimal setup to see how the tool works. It actually seems very simple and trivial to set up. Somehow, I fail to authenticate to my target registry. I must be missing something trivial. Also, the debug output does not help me.

My setup / goal:
Send an image latest from a local on prem registry without authentication to a private namespace in quay.io using a robo account.

My problem / errormessage:
(target obfuscated)
failed to send blob post, ref quay.io/daniel/test:latest: unauthorized

Version

               "maintainer": "",
               "org.opencontainers.image.authors": "regclient contributors",
               "org.opencontainers.image.created": "2024-03-24T18:09:00Z",
               "org.opencontainers.image.description": "regclient/regsync utility for mirroring registry content (scratch)",
               "org.opencontainers.image.documentation": "https://github.com/regclient/regclient",
               "org.opencontainers.image.licenses": "Apache 2.0",
               "org.opencontainers.image.revision": "9de7397da9f1c00dad5213519366002376b8d5ed",
               "org.opencontainers.image.source": "https://github.com/regclient/regclient",
               "org.opencontainers.image.title": "regsync",
               "org.opencontainers.image.url": "https://github.com/regclient/regclient",
               "org.opencontainers.image.vendor": "regclient",
               "org.opencontainers.image.version": "v0.6.0"

Environment

Execution:
docker container run -i --rm -v "$(pwd)/regsync.yml:/home/appuser/regsync.yml:ro" -v "${HOME}/.docker/hub_token:/home/appuser/.docker/hub_token:ro" -e "HUB_USER" ghcr.io/regclient/regsync:latest -v debug -c /home/appuser/regsync.yml once

Platform:
Linux x86_64 RHEL9

My registry:
Local nexus (source)
Remote quay.io (traget)

Setup

Environment

Set username in env variable

# export HUB_USER=daniel+regsync
# echo $HUB_USER 
# daniel+regsync

Hint: Quay automatically adds a + character in the robo's username. Could this be an issue?

Set token in file (obfuscated)

# cat ${HOME}/.docker/hub_token
# AFAS1F8FAENVGTF6AV03HBVY2KDABBEKA734QWODBY6H2U4

Settings

version: 1
creds:
  - registry: local-registry:5000
    tls: disabled
  - registry: quay.io
    user: "{{env \"HUB_USER\"}}"
    pass: "{{file \"/home/appuser/.docker/hub_token\"}}"
defaults:
  ratelimit:
    min: 100
    retry: 15m
  parallel: 2
  interval: 60m
  backup: "bkup-{{.Ref.Tag}}"
sync:
  - source: local-registry:5000/daniel/app/runtime:latest
    target: quay.io/daniel/test/runtime:latest
    type: image

Execution

docker container run -i --rm -v "$(pwd)/regsync.yml:/home/appuser/regsync.yml:ro" -v "${HOME}/.docker/hub_token:/home/appuser/.docker/hub_token:ro" -e "HUB_USER" ghcr.io/regclient/regsync:latest -v debug -c /home/appuser/regsync.yml once

Error

time="2024-05-06T10:34:59+02:00" level=debug msg="http req" method=HEAD url="https://quay.io/v2/daniel/test/runtime/manifests/latest" withAuth=false
time="2024-05-06T10:35:00+02:00" level=debug msg="Auth request parsed" challenge="[{bearer map[realm:https://quay.io/v2/auth scope:repository:daniel/test/runtime:pull service:quay.io]}]"
time="2024-05-06T10:35:00+02:00" level=debug msg="http req" method=HEAD url="https://quay.io/v2/daniel/test/runtime/manifests/latest" withAuth=true
time="2024-05-06T10:35:00+02:00" level=debug msg="Auth request parsed" challenge="[{bearer map[realm:https://quay.io/v2/auth scope:repository:daniel/test/runtime:pull service:quay.io]}]"
time="2024-05-06T10:35:00+02:00" level=debug msg="Failed to handle auth request" Err=unauthorized URL="https://quay.io/v2/daniel/test/runtime/manifests/latest"
...
time="2024-05-06T10:35:01+02:00" level=debug msg="Request failed" URL="https://quay.io/v2/daniel/test/runtime/blobs/uploads/" err="Post \"https://quay.io/v2/daniel/test/runtime/blobs/uploads/\": context canceled"
time="2024-05-06T10:35:01+02:00" level=error msg="Failed to copy image" error="failed to send blob post, ref quay.io/daniel/test/runtime:latest: unauthorized" source="local-registry:5000/daniel/app/runtime:latest" target="quay.io/daniel/test/runtime:latest"
time="2024-05-06T10:35:01+02:00" level=error msg="Failed to sync" error="failed to send blob post, ref quay.io/daniel/test/runtime:latest: unauthorized" source="local-registry:5000/daniel/app/runtime:latest" target="quay.io/daniel/test/runtime:latest"
failed to send blob post, ref quay.io/daniel/test/runtime:latest: unauthorized

Anything else

I did also a test if the login with my quay creds would work and I succeeded. this proves that the environment variable and the token are valid

docker login -u=$HUB_USER -p="mylongtoken" quay.io
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
Login Succeeded!

Any ideas ? :)

[Brandon Mitchell]

I'd try a few things to debug this. First, make sure whitespace or other characters aren't throwing off the file vs the token in the login command with:

docker login -u=$HUB_USER -p="$(cat ${HOME}/.docker/hub_token)" quay.io

Then make sure that token has access to push to the repository with:

docker pull local-registry:5000/daniel/xentis/runtime:latest
docker tag local-registry:5000/daniel/xentis/runtime:latest quay.io/daniel/test/runtime:latest
docker push quay.io/daniel/test/runtime:latest

Then eliminate any interpolation issues by entering the user and secret directly in the yml:

creds:
  - registry: local-registry:5000
    tls: disabled
  - registry: quay.io
    user: "daniel+regsync"
    pass: "yourlongtoken"

If none of that narrows down the issue, include the full debug output that would show the credentials being parsed.

[Athos85]

Good approach. Thanks, I found it, there were two issues and I tested multiple combinations but never this combo. One was my mistake, the other I can't explain.

The first test was successful:

docker login -u=$HUB_USER -p="$(cat ${HOME}/.docker/hub_token)" quay.io
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
Login Succeeded!

Second test:

docker pull ...
docker tag ...
docker push quay.io/daniel/test/runtime:latest
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
Getting image source signatures
Error: trying to reuse blob sha256:31ef0364e9a5089fff79d6ab4a2ccac8398c4add2d838b72e7f5fe1b7a4562 at destination: checking whether a blob sha256:31ef0364e9a5089fff79d6ab4a2ccac8398c4add2d838b72e7f5fe1b7a4562 exists in quay.io/daniel/test/runtime: authentication required

So far so good, this was my bad. In Quay I cannot upload nested directories like in Nexus.

After re-tagging, it worked.

docker tag registry:5000/daniel/app/runtime:latest quay.io/daniel/test:latest

I changed the regsync.yml accordingly.
But the syncer still didn't work.

So I changed the creds as you suggested.

creds:
  - registry: local-registry:5000
    tls: disabled
  - registry: quay.io
    user: "daniel+regsync"
    pass: "yourlongtoken"

It worked!

I narrowed down the problem, it's the line with the pass which did not work. I can add the env variable, but I need to paste the token in plaintext. So this combo works

creds:
  - registry: local-registry:5000
    tls: disabled
  - registry: quay.io
    user: "{{env \"HUB_USER\"}}"
    pass: "yourlongtoken"

Thank you!

[Brandon Mitchell]

If logging in with the token file using cat worked, that points to an issue with the volume mount or, more likely, file permissions (inside the container, regsync is running as uid 1000). If you are running with rootless, or have SELinux enabled, those could also cause access issues.

Overall, the file you are working with contains lots of example configurations, many of which aren't required for all users, so feel free to adjust as needed for your own use case. Docs for the file syntax are available at: https://github.com/regclient/regclient/blob/main/docs/regsync.md