Pushing any container image results in HTTP 500
R0flcopt3r opened this issue · comments
Expected behavior and actual behavior:
Pushing a OCI container to harbor should succeed.
When pushing an image to Harbor hosted in kubernetes via helm results in HTTP error 500.
harbor-core is mentioning this error:
filesystem: unlinkat /storage/docker/registry/v2/repositories/library/alpine/_uploads/81dc943a-b60c-4867-824b-8cc534589664: directory not empty
Steps to reproduce the problem:
deployed via Helm, using chart 1.14.2.
- registry volume is mounted via smb
- all other volumes mounted via locally/nfs using longhorn
it randomly fell into this state. everything was working fine before. Some Longhorn volumes had ran out of space. Extending them did not solve this issue however.
Versions:
Please specify the versions of following systems.
- harbor version: 2.10
- kubernetes version: v1.26.15+rke2r1
Additional context:
harbor yaml configuration only contains OpenID Connect details
logs form harbor-core:
time="2024-05-21T13:59:59.121731631Z" level=info msg="authorized request" go.version=go1.21.4 http.request.host=docker.internal.company.net http.request.id=371f3b19-da8f-4c61-8ec0-bdfcc268c2b8 http.request.method=HEAD http.request.remoteaddr=10.10.50.47 http.request.uri="/v2/library/alpine/blobs/sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8" http.request.useragent="containers/5.29.2 (github.com/containers/image)" vars.digest="sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8" vars.name="library/alpine"
time="2024-05-21T13:59:59.123273367Z" level=info msg="redis: connect harbor-redis:6379" go.version=go1.21.4 instance.id=75600902-1036-4e8c-ae8b-8c93f79bef85 redis.connect.duration=1.314407ms service=registry version=v2.8.3.m
time="2024-05-21T13:59:59.126046129Z" level=error msg="response completed with error" auth.user.name="harbor_registry_user" err.code="blob unknown" err.detail=sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8 err.message="blob unknown to registry" go.version=go1.21.4 http.request.host=docker.internal.company.net http.request.id=371f3b19-da8f-4c61-8ec0-bdfcc268c2b8 http.request.method=HEAD http.request.remoteaddr=10.10.50.47 http.request.uri="/v2/library/alpine/blobs/sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8" http.request.useragent="containers/5.29.2 (github.com/containers/image)" http.response.contenttype="application/json; charset=utf-8" http.response.duration=141.109377ms http.response.status=404 http.response.written=157 vars.digest="sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8" vars.name="library/alpine"
2024-05-21T13:59:59.126126441Z 10.42.221.122 - - [21/May/2024:13:59:58 +0000] "HEAD /v2/library/alpine/blobs/sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8 HTTP/1.1" 404 157 "" "containers/5.29.2 (github.com/containers/image)"
time="2024-05-21T13:59:59.25974Z" level=info msg="authorized request" go.version=go1.21.4 http.request.host=docker.internal.company.net http.request.id=accc7b00-bf7f-4f18-a2ec-e0137ac31570 http.request.method=POST http.request.remoteaddr=10.10.50.47 http.request.uri="/v2/library/alpine/blobs/uploads/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" vars.name="library/alpine"
10.42.221.122 - - [21/May/2024:13:59:59 +0000] "POST /v2/library/alpine/blobs/uploads/ HTTP/1.1" 202 0 "" "containers/5.29.2 (github.com/containers/image)"
2024-05-21T13:59:59.270341978Z time="2024-05-21T13:59:59.270195433Z" level=info msg="response completed" go.version=go1.21.4 http.request.host=docker.internal.company.net http.request.id=accc7b00-bf7f-4f18-a2ec-e0137ac31570 http.request.method=POST http.request.remoteaddr=10.10.50.47 http.request.uri="/v2/library/alpine/blobs/uploads/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" http.response.duration=102.156694ms http.response.status=202 http.response.written=0
time="2024-05-21T13:59:59.434500132Z" level=info msg="authorized request" go.version=go1.21.4 http.request.contenttype="application/octet-stream" http.request.host=docker.internal.company.net http.request.id=81bc64b1-9e21-4e16-915b-031486631e5c http.request.method=PATCH http.request.remoteaddr=10.10.50.47 http.request.uri="/v2/library/alpine/blobs/uploads/81dc943a-b60c-4867-824b-8cc534589664?_state=9MrpgTLFhPTMWvGUxzTbxr8Iix88tCN0Seoq_n5-PtN7Ik5hbWUiOiJsaWJyYXJ5L2FscGluZSIsIlVVSUQiOiI4MWRjOTQzYS1iNjBjLTQ4NjctODI0Yi04Y2M1MzQ1ODk2NjQiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjQtMDUtMjFUMTM6NTk6NTkuMjU5OTU0MzIxWiJ9" http.request.useragent="containers/5.29.2 (github.com/containers/image)" vars.name="library/alpine" vars.uuid=81dc943a-b60c-4867-824b-8cc534589664
time="2024-05-21T13:59:59.493271612Z" level=info msg="response completed" go.version=go1.21.4 http.request.contenttype="application/octet-stream" http.request.host=docker.internal.company.net http.request.id=81bc64b1-9e21-4e16-915b-031486631e5c http.request.method=PATCH http.request.remoteaddr=10.10.50.47 http.request.uri="/v2/library/alpine/blobs/uploads/81dc943a-b60c-4867-824b-8cc534589664?_state=9MrpgTLFhPTMWvGUxzTbxr8Iix88tCN0Seoq_n5-PtN7Ik5hbWUiOiJsaWJyYXJ5L2FscGluZSIsIlVVSUQiOiI4MWRjOTQzYS1iNjBjLTQ4NjctODI0Yi04Y2M1MzQ1ODk2NjQiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjQtMDUtMjFUMTM6NTk6NTkuMjU5OTU0MzIxWiJ9" http.request.useragent="containers/5.29.2 (github.com/containers/image)" http.response.duration=147.125438ms http.response.status=202 http.response.written=0
2024-05-21T13:59:59.493468014Z 10.42.221.122 - - [21/May/2024:13:59:59 +0000] "PATCH /v2/library/alpine/blobs/uploads/81dc943a-b60c-4867-824b-8cc534589664?_state=9MrpgTLFhPTMWvGUxzTbxr8Iix88tCN0Seoq_n5-PtN7Ik5hbWUiOiJsaWJyYXJ5L2FscGluZSIsIlVVSUQiOiI4MWRjOTQzYS1iNjBjLTQ4NjctODI0Yi04Y2M1MzQ1ODk2NjQiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjQtMDUtMjFUMTM6NTk6NTkuMjU5OTU0MzIxWiJ9 HTTP/1.1" 202 0 "" "containers/5.29.2 (github.com/containers/image)"
10.42.221.122 - - [21/May/2024:13:59:59 +0000] "GET / HTTP/1.1" 200 0 "" "Go-http-client/1.1"
time="2024-05-21T13:59:59.617429276Z" level=info msg="authorized request" go.version=go1.21.4 http.request.contenttype="application/octet-stream" http.request.host=docker.internal.company.net http.request.id=c7e3d048-d761-4422-abb3-432a3b91b326 http.request.method=PUT http.request.remoteaddr=10.10.50.47 http.request.uri="/v2/library/alpine/blobs/uploads/81dc943a-b60c-4867-824b-8cc534589664?_state=1Q-OlHNkv-ghZkRZ9kePBwnZre9VRFv18xqCrQe2Tz57Ik5hbWUiOiJsaWJyYXJ5L2FscGluZSIsIlVVSUQiOiI4MWRjOTQzYS1iNjBjLTQ4NjctODI0Yi04Y2M1MzQ1ODk2NjQiLCJPZmZzZXQiOjM1MjkxMDMsIlN0YXJ0ZWRBdCI6IjIwMjQtMDUtMjFUMTM6NTk6NTlaIn0%3D&digest=sha256%3Aa0bed814693a2470db647d234b2a32f9b31153fffa71e1dad78db9166a1428a4" http.request.useragent="containers/5.29.2 (github.com/containers/image)" vars.name="library/alpine" vars.uuid=81dc943a-b60c-4867-824b-8cc534589664
time="2024-05-21T13:59:59.632046884Z" level=error msg="unable to delete layer upload resources "/docker/registry/v2/repositories/library/alpine/_uploads/81dc943a-b60c-4867-824b-8cc534589664": filesystem: unlinkat /storage/docker/registry/v2/repositories/library/alpine/_uploads/81dc943a-b60c-4867-824b-8cc534589664: directory not empty" auth.user.name="harbor_registry_user" go.version=go1.21.4 http.request.contenttype="application/octet-stream" http.request.host=docker.internal.company.net http.request.id=c7e3d048-d761-4422-abb3-432a3b91b326 http.request.method=PUT http.request.remoteaddr=10.10.50.47 http.request.uri="/v2/library/alpine/blobs/uploads/81dc943a-b60c-4867-824b-8cc534589664?_state=1Q-OlHNkv-ghZkRZ9kePBwnZre9VRFv18xqCrQe2Tz57Ik5hbWUiOiJsaWJyYXJ5L2FscGluZSIsIlVVSUQiOiI4MWRjOTQzYS1iNjBjLTQ4NjctODI0Yi04Y2M1MzQ1ODk2NjQiLCJPZmZzZXQiOjM1MjkxMDMsIlN0YXJ0ZWRBdCI6IjIwMjQtMDUtMjFUMTM6NTk6NTlaIn0%3D&digest=sha256%3Aa0bed814693a2470db647d234b2a32f9b31153fffa71e1dad78db9166a1428a4" http.request.useragent="containers/5.29.2 (github.com/containers/image)" vars.name="library/alpine" vars.uuid=81dc943a-b60c-4867-824b-8cc534589664
time="2024-05-21T13:59:59.632116688Z" level=error msg="unknown error completing upload: filesystem: unlinkat /storage/docker/registry/v2/repositories/library/alpine/_uploads/81dc943a-b60c-4867-824b-8cc534589664: directory not empty" auth.user.name="harbor_registry_user" go.version=go1.21.4 http.request.contenttype="application/octet-stream" http.request.host=docker.internal.company.net http.request.id=c7e3d048-d761-4422-abb3-432a3b91b326 http.request.method=PUT http.request.remoteaddr=10.10.50.47 http.request.uri="/v2/library/alpine/blobs/uploads/81dc943a-b60c-4867-824b-8cc534589664?_state=1Q-OlHNkv-ghZkRZ9kePBwnZre9VRFv18xqCrQe2Tz57Ik5hbWUiOiJsaWJyYXJ5L2FscGluZSIsIlVVSUQiOiI4MWRjOTQzYS1iNjBjLTQ4NjctODI0Yi04Y2M1MzQ1ODk2NjQiLCJPZmZzZXQiOjM1MjkxMDMsIlN0YXJ0ZWRBdCI6IjIwMjQtMDUtMjFUMTM6NTk6NTlaIn0%3D&digest=sha256%3Aa0bed814693a2470db647d234b2a32f9b31153fffa71e1dad78db9166a1428a4" http.request.useragent="containers/5.29.2 (github.com/containers/image)" vars.name="library/alpine" vars.uuid=81dc943a-b60c-4867-824b-8cc534589664
2024-05-21T13:59:59.632319945Z time="2024-05-21T13:59:59.632161776Z" level=error msg="error canceling upload after error: already closed" auth.user.name="harbor_registry_user" go.version=go1.21.4 http.request.contenttype="application/octet-stream" http.request.host=docker.internal.company.net http.request.id=c7e3d048-d761-4422-abb3-432a3b91b326 http.request.method=PUT http.request.remoteaddr=10.10.50.47 http.request.uri="/v2/library/alpine/blobs/uploads/81dc943a-b60c-4867-824b-8cc534589664?_state=1Q-OlHNkv-ghZkRZ9kePBwnZre9VRFv18xqCrQe2Tz57Ik5hbWUiOiJsaWJyYXJ5L2FscGluZSIsIlVVSUQiOiI4MWRjOTQzYS1iNjBjLTQ4NjctODI0Yi04Y2M1MzQ1ODk2NjQiLCJPZmZzZXQiOjM1MjkxMDMsIlN0YXJ0ZWRBdCI6IjIwMjQtMDUtMjFUMTM6NTk6NTlaIn0%3D&digest=sha256%3Aa0bed814693a2470db647d234b2a32f9b31153fffa71e1dad78db9166a1428a4" http.request.useragent="containers/5.29.2 (github.com/containers/image)" vars.name="library/alpine" vars.uuid=81dc943a-b60c-4867-824b-8cc534589664
10.42.221.122 - - [21/May/2024:13:59:59 +0000] "PUT /v2/library/alpine/blobs/uploads/81dc943a-b60c-4867-824b-8cc534589664?_state=1Q-OlHNkv-ghZkRZ9kePBwnZre9VRFv18xqCrQe2Tz57Ik5hbWUiOiJsaWJyYXJ5L2FscGluZSIsIlVVSUQiOiI4MWRjOTQzYS1iNjBjLTQ4NjctODI0Yi04Y2M1MzQ1ODk2NjQiLCJPZmZzZXQiOjM1MjkxMDMsIlN0YXJ0ZWRBdCI6IjIwMjQtMDUtMjFUMTM6NTk6NTlaIn0%3D&digest=sha256%3Aa0bed814693a2470db647d234b2a32f9b31153fffa71e1dad78db9166a1428a4 HTTP/1.1" 500 244 "" "containers/5.29.2 (github.com/containers/image)"
time="2024-05-21T13:59:59.634421681Z" level=error msg="response completed with error" auth.user.name="harbor_registry_user" err.code=unknown err.detail="filesystem: unlinkat /storage/docker/registry/v2/repositories/library/alpine/_uploads/81dc943a-b60c-4867-824b-8cc534589664: directory not empty" err.message="unknown error" go.version=go1.21.4 http.request.contenttype="application/octet-stream" http.request.host=docker.internal.company.net http.request.id=c7e3d048-d761-4422-abb3-432a3b91b326 http.request.method=PUT http.request.remoteaddr=10.10.50.47 http.request.uri="/v2/library/alpine/blobs/uploads/81dc943a-b60c-4867-824b-8cc534589664?_state=1Q-OlHNkv-ghZkRZ9kePBwnZre9VRFv18xqCrQe2Tz57Ik5hbWUiOiJsaWJyYXJ5L2FscGluZSIsIlVVSUQiOiI4MWRjOTQzYS1iNjBjLTQ4NjctODI0Yi04Y2M1MzQ1ODk2NjQiLCJPZmZzZXQiOjM1MjkxMDMsIlN0YXJ0ZWRBdCI6IjIwMjQtMDUtMjFUMTM6NTk6NTlaIn0%3D&digest=sha256%3Aa0bed814693a2470db647d234b2a32f9b31153fffa71e1dad78db9166a1428a4" http.request.useragent="containers/5.29.2 (github.com/containers/image)" http.response.contenttype="application/json; charset=utf-8" http.response.duration=104.92379ms http.response.status=500 http.response.written=244 vars.name="library/alpine" vars.uuid=81dc943a-b60c-4867-824b-8cc534589664
10.10.0.144 - - [21/May/2024:13:59:59 +0000] "GET / HTTP/1.1" 200 0 "" "kube-probe/1.26"
2024-05-21T13:59:59.847531821Z 10.10.0.144 - - [21/May/2024:13:59:59 +0000] "GET / HTTP/1.1" 200 0 "" "kube-probe/1.26"
values.json looks like this:
{
"caSecretName": "",
"cache": {
"enabled": false,
"expireHours": 24
},
"core": {
"affinity": {},
"artifactPullAsyncFlushDuration": null,
"automountServiceAccountToken": false,
"configureUserSettings": "[REDACTED]",
"existingSecret": "",
"existingXsrfSecret": "",
"existingXsrfSecretKey": "CSRF_KEY",
"extraEnvVars": [],
"gdpr": {
"deleteUser": false
},
"image": {
"repository": "goharbor/harbor-core",
"tag": "v2.10.0"
},
"nodeSelector": {},
"podAnnotations": {},
"podLabels": {},
"priorityClassName": null,
"quotaUpdateProvider": "db",
"replicas": 1,
"revisionHistoryLimit": 10,
"secret": "",
"secretName": "",
"serviceAccountName": "",
"serviceAnnotations": {},
"startupProbe": {
"enabled": true,
"initialDelaySeconds": 10
},
"tokenCert": "",
"tokenKey": "",
"tolerations": [],
"topologySpreadConstraints": [],
"xsrfKey": ""
},
"database": {
"external": {
"coreDatabase": "registry",
"existingSecret": "",
"host": "192.168.0.1",
"password": "[REDACTED]",
"port": "5432",
"sslmode": "disable",
"username": "[REDACTED]"
},
"internal": {
"affinity": {},
"automountServiceAccountToken": false,
"extraEnvVars": [],
"image": {
"repository": "goharbor/harbor-db",
"tag": "v2.10.0"
},
"initContainer": {
"migrator": {},
"permissions": {}
},
"livenessProbe": {
"timeoutSeconds": 1
},
"nodeSelector": {},
"password": "[REDACTED]",
"priorityClassName": null,
"readinessProbe": {
"timeoutSeconds": 1
},
"serviceAccountName": "",
"shmSizeLimit": "512Mi",
"tolerations": []
},
"maxIdleConns": 100,
"maxOpenConns": 900,
"podAnnotations": {},
"podLabels": {},
"type": "internal"
},
"enableMigrateHelmHook": false,
"existingSecretAdminPassword": "admin-password",
"existingSecretAdminPasswordKey": "HARBOR_ADMIN_PASSWORD",
"existingSecretSecretKey": "",
"exporter": {
"affinity": {},
"automountServiceAccountToken": false,
"cacheCleanInterval": 14400,
"cacheDuration": 23,
"extraEnvVars": [],
"image": {
"repository": "goharbor/harbor-exporter",
"tag": "v2.10.0"
},
"nodeSelector": {},
"podAnnotations": {},
"podLabels": {},
"priorityClassName": null,
"replicas": 1,
"revisionHistoryLimit": 10,
"serviceAccountName": "",
"tolerations": [],
"topologySpreadConstraints": []
},
"expose": {
"clusterIP": {
"annotations": {},
"name": "harbor",
"ports": {
"httpPort": 80,
"httpsPort": 443
},
"staticClusterIP": ""
},
"ingress": {
"annotations": {
"ingress.kubernetes.io/proxy-body-size": "0",
"ingress.kubernetes.io/ssl-redirect": "true",
"nginx.ingress.kubernetes.io/proxy-body-size": "0",
"nginx.ingress.kubernetes.io/ssl-redirect": "true"
},
"className": "",
"controller": "default",
"harbor": {
"annotations": {},
"labels": {}
},
"hosts": {
"core": "docker.internal.appear.net"
},
"kubeVersionOverride": ""
},
"loadBalancer": {
"IP": "",
"annotations": {},
"name": "harbor",
"ports": {
"httpPort": 80,
"httpsPort": 443
},
"sourceRanges": []
},
"nodePort": {
"name": "harbor",
"ports": {
"http": {
"nodePort": 30002,
"port": 80
},
"https": {
"nodePort": 30003,
"port": 443
}
}
},
"tls": {
"certSource": "secret",
"enabled": true,
"secret": {
"secretName": "kube-internal-company-net-wildcard"
}
},
"type": "ingress"
},
"externalURL": "https://docker.internal.company.net",
"imagePullPolicy": "IfNotPresent",
"imagePullSecrets": null,
"internalTLS": {
"certSource": "auto",
"core": {
"crt": "",
"key": "",
"secretName": ""
},
"enabled": false,
"jobservice": {
"crt": "",
"key": "",
"secretName": ""
},
"portal": {
"crt": "",
"key": "",
"secretName": ""
},
"registry": {
"crt": "",
"key": "",
"secretName": ""
},
"strong_ssl_ciphers": false,
"trivy": {
"crt": "",
"key": "",
"secretName": ""
},
"trustCa": ""
},
"ipFamily": {
"ipv4": {
"enabled": true
},
"ipv6": {
"enabled": true
}
},
"jobservice": {
"affinity": {},
"automountServiceAccountToken": false,
"existingSecret": "",
"existingSecretKey": "JOBSERVICE_SECRET",
"extraEnvVars": [],
"image": {
"repository": "goharbor/harbor-jobservice",
"tag": "v2.10.0"
},
"jobLoggers": [
"file"
],
"loggerSweeperDuration": 14,
"maxJobWorkers": 10,
"nodeSelector": {},
"notification": {
"webhook_job_http_client_timeout": 3,
"webhook_job_max_retry": 3
},
"podAnnotations": {},
"podLabels": {},
"priorityClassName": null,
"reaper": {
"max_dangling_hours": 168,
"max_update_hours": 24
},
"replicas": 1,
"revisionHistoryLimit": 10,
"secret": "",
"serviceAccountName": "",
"tolerations": [],
"topologySpreadConstraints": null
},
"logLevel": "info",
"metrics": {
"core": {
"path": "/metrics",
"port": 8001
},
"enabled": false,
"exporter": {
"path": "/metrics",
"port": 8001
},
"jobservice": {
"path": "/metrics",
"port": 8001
},
"registry": {
"path": "/metrics",
"port": 8001
},
"serviceMonitor": {
"additionalLabels": {},
"enabled": false,
"interval": "",
"metricRelabelings": [],
"relabelings": []
}
},
"nginx": {
"affinity": {},
"automountServiceAccountToken": false,
"extraEnvVars": [],
"image": {
"repository": "goharbor/nginx-photon",
"tag": "v2.10.0"
},
"nodeSelector": {},
"podAnnotations": {},
"podLabels": {},
"priorityClassName": null,
"replicas": 1,
"revisionHistoryLimit": 10,
"serviceAccountName": "",
"tolerations": [],
"topologySpreadConstraints": []
},
"persistence": {
"enabled": true,
"imageChartStorage": {
"azure": {
"accountkey": "base64encodedaccountkey",
"accountname": "accountname",
"container": "containername",
"existingSecret": ""
},
"disableredirect": false,
"filesystem": {
"rootdirectory": "/storage"
},
"gcs": {
"bucket": "bucketname",
"encodedkey": "base64-encoded-json-key-file",
"existingSecret": "",
"useWorkloadIdentity": false
},
"oss": {
"accesskeyid": "[REDACTED]",
"accesskeysecret": "[REDACTED]",
"bucket": "bucketname",
"existingSecret": "",
"region": "regionname"
},
"s3": {
"bucket": "bucketname",
"region": "us-west-1"
},
"swift": {
"authurl": "https://storage.myprovider.com/v3/auth",
"container": "containername",
"existingSecret": "",
"password": "[REDACTED]",
"username": "[REDACTED]"
},
"type": "filesystem"
},
"persistentVolumeClaim": {
"database": {
"accessMode": "ReadWriteOnce",
"existingClaim": "harbor-database",
"size": "10Gi",
"storageClass": "longhorn"
},
"jobservice": {
"jobLog": {
"accessMode": "ReadWriteOnce",
"existingClaim": "harbor-job-service",
"size": "10Gi",
"storageClass": "longhorn"
}
},
"redis": {
"accessMode": "ReadWriteOnce",
"existingClaim": "harbor-redis",
"size": "10Gi",
"storageClass": "longhorn"
},
"registry": {
"accessMode": "ReadWriteOnce",
"existingClaim": "harbor-registry",
"size": "2Ti",
"storageClass": "smb"
},
"trivy": {
"accessMode": "ReadWriteOnce",
"existingClaim": "harbor-trivy",
"size": "50Gi",
"storageClass": "longhorn"
}
},
"resourcePolicy": "keep"
},
"portal": {
"affinity": {},
"automountServiceAccountToken": false,
"extraEnvVars": [],
"image": {
"repository": "goharbor/harbor-portal",
"tag": "v2.10.0"
},
"nodeSelector": {},
"podAnnotations": {},
"podLabels": {},
"priorityClassName": null,
"replicas": 1,
"revisionHistoryLimit": 10,
"serviceAccountName": "",
"serviceAnnotations": {},
"tolerations": [],
"topologySpreadConstraints": []
},
"proxy": {
"components": [
"core",
"jobservice",
"trivy"
],
"httpProxy": null,
"httpsProxy": null,
"noProxy": "127.0.0.1,localhost,.local,.internal"
},
"redis": {
"external": {
"addr": "192.168.0.2:6379",
"coreDatabaseIndex": "0",
"existingSecret": "",
"jobserviceDatabaseIndex": "1",
"password": "",
"registryDatabaseIndex": "2",
"sentinelMasterSet": "",
"trivyAdapterIndex": "5",
"username": ""
},
"internal": {
"affinity": {},
"automountServiceAccountToken": false,
"extraEnvVars": [],
"image": {
"repository": "goharbor/redis-photon",
"tag": "v2.10.0"
},
"jobserviceDatabaseIndex": "1",
"nodeSelector": {},
"priorityClassName": null,
"registryDatabaseIndex": "2",
"serviceAccountName": "",
"tolerations": [],
"trivyAdapterIndex": "5"
},
"podAnnotations": {},
"podLabels": {},
"type": "internal"
},
"registry": {
"affinity": {},
"automountServiceAccountToken": false,
"controller": {
"extraEnvVars": [],
"image": {
"repository": "goharbor/harbor-registryctl",
"tag": "v2.10.0"
}
},
"credentials": {
"existingSecret": "",
"htpasswdString": "",
"password": "[REDACTED]",
"username": "[REDACTED]"
},
"existingSecret": "",
"existingSecretKey": "REGISTRY_HTTP_SECRET",
"middleware": {
"cloudFront": {
"baseurl": "example.cloudfront.net",
"duration": "3000s",
"ipfilteredby": "none",
"keypairid": "KEYPAIRID",
"privateKeySecret": "my-secret"
},
"enabled": false,
"type": "cloudFront"
},
"nodeSelector": {},
"podAnnotations": {},
"podLabels": {},
"priorityClassName": null,
"registry": {
"extraEnvVars": [],
"image": {
"repository": "goharbor/registry-photon",
"tag": "v2.10.0"
}
},
"relativeurls": false,
"replicas": 1,
"revisionHistoryLimit": 10,
"secret": "",
"serviceAccountName": "",
"tolerations": [],
"topologySpreadConstraints": [],
"upload_purging": {
"age": "168h",
"dryrun": false,
"enabled": true,
"interval": "24h"
}
},
"secretKey": "[REDACTED]",
"trace": {
"enabled": false,
"jaeger": {
"endpoint": "http://hostname:14268/api/traces"
},
"otel": {
"compression": false,
"endpoint": "hostname:4318",
"insecure": true,
"timeout": 10,
"url_path": "/v1/traces"
},
"provider": "jaeger",
"sample_rate": 1
},
"trivy": {
"affinity": {},
"automountServiceAccountToken": false,
"debugMode": false,
"enabled": true,
"extraEnvVars": [],
"gitHubToken": "",
"ignoreUnfixed": false,
"image": {
"repository": "goharbor/trivy-adapter-photon",
"tag": "v2.10.0"
},
"insecure": false,
"nodeSelector": {},
"offlineScan": false,
"podAnnotations": {},
"podLabels": {},
"priorityClassName": null,
"replicas": 1,
"resources": {
"limits": {
"cpu": 1,
"memory": "1Gi"
},
"requests": {
"cpu": "200m",
"memory": "512Mi"
}
},
"securityCheck": "vuln",
"serviceAccountName": "",
"severity": "UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL",
"skipUpdate": false,
"timeout": "5m0s",
"tolerations": [],
"topologySpreadConstraints": [],
"vulnType": "os,library"
},
"updateStrategy": {
"type": "RollingUpdate"
}
}
Thankfully we haven't migrated to Harbor for production, so we can rebuild the data if we need to, but this doesn't inspire confidence if we can't find a way to at the very least recover the instance. Going to try to do that now, any tips would be greatly appreciated. We have terrabytes of images.. Some very old that sadly can't be reproduced.
Have your registry pv (folder) been mounted/shared to other process as well?
I tried to so some manual clean up. deleting the files harbor were complaining about manually, removing the repository i tried to push, all without any success.
Is there a way to recover the registry folder with a fresh database? If i spin up a new harbor instance with a copy of the registry folder from the broken instance with a fresh database volume harbor is empty.
Have your registry pv (folder) been mounted/shared to other process as well?
no, but it is mounted via smb on some HA/failover system that might have had some hiccups where it has performed failover a couple times. It is possible that a container image were being written to the registry as a failover took place. This usually spikes disk latency by a lot, and might give a filesystem error if things time out. If that is the root cause, it still feels weird that I can't push any image anymore to any project.
Is this instance a production one or just a testbed?
If it's a testbed without any important image/data, we could testify against it by re-claim a volume of registry using other storage class to determine whether there's a written issue of smb mounted volume
Seems it would be something unfinished uploads between distribution and filesystem(smb), could you refer this comment
- to ls what's in the
/storage/docker/registry/v2/repositories/library/alpine/_uploads/81dc943a-b60c-4867-824b-8cc534589664 - try to remove
hashstatesdata if there's any
contents looks like this:
~ > ls /mnt/k8s_pool/harbor-registry/docker/registry/v2/repositories/library/alpine/_uploads/0898bc01-a49e-4215-9013-ec4e57034452
hashstates
~ > tree /mnt/k8s_pool/harbor-registry/docker/registry/v2/repositories/library/alpine/_uploads/0898bc01-a49e-4215-9013-ec4e57034452
/mnt/k8s_pool/harbor-registry/docker/registry/v2/repositories/library/alpine/_uploads/0898bc01-a49e-4215-9013-ec4e57034452
└── hashstates
└── sha256
├── 0
└── 3529103
I earlier tried to delete the contents of _uploads. both by deleting the entire _uploads folder, and by systematically deleting the contents. It just gets replaced by a new UUID, and i get the same error.
I just checked dmesg on the node running the harbor pods and it is giving me this when i try to push:
[1962593.416490] CIFS: VFS: directory entry name would overflow frame end of buf 000000003e975ef6
so it seems smb is indeed the culprit here. I'm going to investigate some more and get back to you.
This took a while. After downgrading from flatcar 3815.2.2 to 3815.2.0 we no longer see any issues with Harbor. problem was indeed with CIFS bugging out. I don't think Harbor could have been more helpful figuring out what the problem was.