missing docker images: http status code: 404

[SebastianStrajan]

Expected behavior and actual behavior:
We upgraded from a single instance older version of harbor with CIFS storage to a distributed harbor setup deployed inside kubernetes with S3 storage and now we are getting complaints that images are missing randomly, especially when k8s clusters are upgraded and older apps that haven't been deployed for a while are redeployed.

When I go to a image with problems I can't scan it or see the build layers if I click the Build History button I get the following:
http status code: 404, body: {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Name":"dummysample/dummysample-bla-bla","Revision":"sha256:4d727.....76e93e"}}]}

Steps to reproduce the problem:
Don't know what is causing this to be able to reproduce. Found multiple cases with a similar issue for harbor but when they tried to do deletes/push requests. We want to avoid losing images.

Not sure if this is caused by harbor GC or redis or due to the S3 storage that we are using (a few weeks back versioning was activated and we are planning on checking if the layers are still there to be sure there aren't any issues with the storage part on our end).

Versions:
Please specify the versions of following systems.

• harbor version: Version v2.7.1-6015b3ef

Additional context:
Error response from daemon: manifest for registry-domain/dummysample/dummysample-bla-bla:1 not found: manifest unknown: manifest unknown

Any ideas why this is happening?

Thanks,
Sebastian

[MinerYang]

Looks like during the transfer from CIFS storage to S3 storage, the files in registry volume are not persistent. You could check inside the registry container the existence of this manifest

kubectl exec -it <harbor-registry>  -c registry -n <namespace> -- /bin/bash
cat /storage/docker/registry/v2/blobs/sha256/4d/4d727.....76e93e/data

[SebastianStrajan]

We did not do any raw transfer. Most images have been pushed in both registries during the last 6 months, so there shouldn't be any corruptions cause by this. And for a couple of projects there was a replication done prior to the transition to ensure all images are there.

Found problems for both types of 'transfers'. We are considering an harbor upgrade but not sure if this will be a fix.