Verify by key fails but verify by hashivault is success

Question

Verify by key fails but verify by hashivault is success

GarrykZ opened this issue 7 months ago · comments

Describe the bug
I'm configuring sign-check in k8s with kyverno but stopped by an issue when my sign didnt check properly.
When I started checking this behavior i found next:
root@srv:/home# cosign verify --key hashivault://cosign harbor.domain.local/test/tomcat@sha256:b74dd6880fb1cd194fe23f3d2bec0cd0e89e79b609302632eb666c09c9e77709 --insecure-ignore-tlog=true --insecure-ignore-sct=true
and got:

Verification for harbor.domain.local/test/tomcat@sha256:b74dd6880fb1cd194fe23f3d2bec0cd0e89e79b609302632eb666c09c9e77709 --
The following checks were performed on each of these signatures:

The cosign claims were validated
The signatures were verified against the specified public key

But if i do this:
root@srv:/home# cosign public-key --key hashivault://cosign > key.key
root@srv:/home# cosign verify --key key.key harbor.domain.local/test/tomcat@sha256:b74dd6880fb1cd194fe23f3d2bec0cd0e89e79b609302632eb666c09c9e77709 --insecure-ignore-tlog=true --insecure-ignore-sct=true
I got:
Error: no matching signatures: crypto/rsa: verification error
main.go:69: error during command execution: no matching signatures: crypto/rsa: verification error

When I looking logs in debug-mode i didnt find any clear error, last step - checking blob and answers are the same when checking by key and by vault.

To Reproduce

Create keys in vault
Sign image by vault
Get public-key from vault
Try to verify image by local public-key

Expected behavior
Veryfing image by key from vault and by local key is successfull

Console Output

Debug by local key:

root@srv:/home# cosign verify --key key.key harbor.domain.local/kb/test/tomcat@sha256:b74dd6880fb1cd194fe23f3d2bec0cd0e89e79b609302632eb666c09c9e77709 --insecure-ignore-tlog=true --insecure-ignore-sct=true -d WARNING: Skipping tlog verification is an insecure practice that lacks of transparency and auditability verification for the signature. 2023/11/16 11:48:50 --> GET https://harbor.domain.local/v2/ 2023/11/16 11:48:50 GET /v2/ HTTP/1.1 Host: harbor.domain.local User-Agent: cosign/v2.1.1 (linux; amd64) go-containerregistry/v0.15.2 Accept-Encoding: gzip

2023/11/16 11:48:50 <-- 401 https://harbor.domain.local/v2/ (96.826479ms)
2023/11/16 11:48:50 HTTP/1.1 401 Unauthorized
Content-Length: 76
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Thu, 16 Nov 2023 08:48:50 GMT
Docker-Distribution-Api-Version: registry/2.0
Server: nginx
Set-Cookie: sid=d058ab4dadb8e1f794d971fcf0aec0f9; Path=/; HttpOnly
Www-Authenticate: Bearer realm="https://harbor.domain.local/service/token",service="harbor-registry"
X-Request-Id: 722e944b-6415-46cb-96f1-acdde8fad03f

{"errors":[{"code":"UNAUTHORIZED","message":"unauthorized: unauthorized"}]}

2023/11/16 11:48:50 --> GET https://harbor.domain.local/service/token?scope=repository%3Akb%2Ftest%2Ftomcat%3Apull&service=harbor-registry [body redacted: basic token response contains credentials]
2023/11/16 11:48:50 GET /service/token?scope=repository%3Akb%2Ftest%2Ftomcat%3Apull&service=harbor-registry HTTP/1.1
Host: harbor.domain.local
User-Agent: cosign/v2.1.1 (linux; amd64) go-containerregistry/v0.15.2
Authorization:
Accept-Encoding: gzip

2023/11/16 11:48:51 <-- 200 https://harbor.domain.local/service/token?scope=repository%3Akb%2Ftest%2Ftomcat%3Apull&service=harbor-registry (49.25939ms) [body redacted: basic token response contains credentials]
2023/11/16 11:48:51 HTTP/1.1 200 OK
Connection: keep-alive
Content-Security-Policy: frame-ancestors 'none'
Content-Type: application/json; charset=utf-8
Date: Thu, 16 Nov 2023 08:48:51 GMT
Server: nginx
Set-Cookie: sid=0adde6f2e3a48214a796a437a036be45; Path=/; Secure; HttpOnly
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Frame-Options: DENY
X-Request-Id: bfcce794-4a03-4209-90db-1246bf2ab6a6

2023/11/16 11:48:51 --> GET https://harbor.domain.local/v2/test/tomcat/referrers/sha256:b74dd6880fb1cd194fe23f3d2bec0cd0e89e79b609302632eb666c09c9e77709
2023/11/16 11:48:51 GET /v2/test/tomcat/referrers/sha256:b74dd6880fb1cd194fe23f3d2bec0cd0e89e79b609302632eb666c09c9e77709 HTTP/1.1
Host: harbor.domain.local
User-Agent: cosign/v2.1.1 (linux; amd64) go-containerregistry/v0.15.2
Accept: application/vnd.oci.image.index.v1+json
Authorization:
Accept-Encoding: gzip

2023/11/16 11:48:51 <-- 200 https://harbor.domain.local/v2/test/tomcat/referrers/sha256:b74dd6880fb1cd194fe23f3d2bec0cd0e89e79b609302632eb666c09c9e77709 (10.297905ms)
2023/11/16 11:48:51 HTTP/1.1 200 OK
Content-Length: 300
Connection: keep-alive
Content-Security-Policy: frame-ancestors 'none'
Content-Type: application/json; charset=utf-8
Date: Thu, 16 Nov 2023 08:48:51 GMT
Server: nginx
Set-Cookie: sid=8b112d1caa3e784b91f139ec298c0fd5; Path=/; HttpOnly
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Frame-Options: DENY
X-Request-Id: 20d2a502-4122-49ea-9186-ac1ccc5c2820
X-Total-Count: 1

{"schemaVersion":2,"mediaType":"application/vnd.oci.image.index.v1+json","manifests":[{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:b45b2da7cc36031dc256469e621f56dece4fbf7e2bda3559920c83af35267d69","size":1641,"artifactType":"application/vnd.oci.image.config.v1+json"}]}

2023/11/16 11:48:51 --> GET https://harbor.domain.local/v2/test/tomcat/manifests/sha256-b74dd6880fb1cd194fe23f3d2bec0cd0e89e79b609302632eb666c09c9e77709.sig
2023/11/16 11:48:51 GET /v2/test/tomcat/manifests/sha256-b74dd6880fb1cd194fe23f3d2bec0cd0e89e79b609302632eb666c09c9e77709.sig HTTP/1.1
Host: harbor.domain.local
User-Agent: cosign/v2.1.1 (linux; amd64) go-containerregistry/v0.15.2
Accept: application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json
Authorization:
Accept-Encoding: gzip

2023/11/16 11:48:51 <-- 200 https://harbor.domain.local/v2/test/tomcat/manifests/sha256-b74dd6880fb1cd194fe23f3d2bec0cd0e89e79b609302632eb666c09c9e77709.sig (21.477203ms)
2023/11/16 11:48:51 HTTP/1.1 200 OK
Content-Length: 1146
Connection: keep-alive
Content-Security-Policy: frame-ancestors 'none'
Content-Type: application/vnd.oci.image.manifest.v1+json
Date: Thu, 16 Nov 2023 08:48:51 GMT
Docker-Content-Digest: sha256:b45b2da7cc36031dc256469e621f56dece4fbf7e2bda3559920c83af35267d69
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:b45b2da7cc36031dc256469e621f56dece4fbf7e2bda3559920c83af35267d69"
Server: nginx
Set-Cookie: sid=55b666ead8189b5cf99e8d9348551ef8; Path=/; HttpOnly
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Frame-Options: DENY
X-Request-Id: 189325e5-5b21-4431-b6b7-c1881b485070

{"schemaVersion":2,"mediaType":"application/vnd.oci.image.manifest.v1+json","config":{"mediaType":"application/vnd.oci.image.config.v1+json","size":248,"digest":"sha256:8017b161b26b330beec138cc4bd529021c76c7e8d2bc360c52bd0e90bdad9461"},"layers":[{"mediaType":"application/vnd.dev.cosign.simplesigning.v1+json","size":247,"digest":"sha256:edc06179faadbe516edba2e5b91d37e992ca0cb19c1f79a2ee2862253d0ba7b3","annotations":{"dev.cosignproject.cosign/signature":"kC6ENoScap3Du8cfjs0pjxaRlbN6hDhjn0KlPQpwdnoLwSzluD7BJUwJdRh2VV9NzqT1SJ0TThGl50iIOXkqJWdFxNJP+cCPWC46ZX6zhSRemkSBAOqjELsRGWFNV5RMkPQaG+jtdV59h033cKIkhuZGOeCJa0WNF7uc7+eBdj63TrtwC1RtP0tcWSdeUvYhTZJXamyxKFGQub3B6vL4z5QL04c7Yy6DmlyFDZLhs1Fq4c7X+a2OOTSTjaDJJuMPxxFO7EvJ+hD7uIbEDUOzjb+KLfCaKT6zZxFfIgihDHzx79zA5ob6fMwquZNXWtsBapTEhZx4DDtuvkJYw0Xz6+tnQlZ7Im8THUaWqOwu6Yjgn0pFr7LRrF3Ffcz2c63hFHFPquzOTKj2A8G4EbEXA+Avvj5ERoomFFxQvB1w+pph1D0fsmXuVjkGTm0/zEkZH73tzjQ34dbi8uRtawJxXMZ2C8A0nZPIv6WpvV/P7lJjFd6vXerZVUD5lXLTXopEu/re5W1b+k8UAkYhIfh1GPBxzsIo2HKcgc3As37GqAVfXE9Jebu0GEAr8h2mCVjg11rsoA+d6tELnajhRKcaK8jmk0csu/9xVlfLOJTryhNKgDKKZ9j3WJ41YybrKR2r/gTfiz8EDgRAnS1J2e+MAPdOgNXWVf8Un6LE3C8lbk8="}}]}
2023/11/16 11:48:51 --> GET https://harbor.domain.local/v2/test/tomcat/blobs/sha256:edc06179faadbe516edba2e5b91d37e992ca0cb19c1f79a2ee2862253d0ba7b3 [body redacted: omitting binary blobs from logs]
2023/11/16 11:48:51 GET /v2/test/tomcat/blobs/sha256:edc06179faadbe516edba2e5b91d37e992ca0cb19c1f79a2ee2862253d0ba7b3 HTTP/1.1
Host: harbor.domain.local
User-Agent: cosign/v2.1.1 (linux; amd64) go-containerregistry/v0.15.2
Authorization:
Accept-Encoding: gzip

2023/11/16 11:48:51 <-- 200 https://harbor.domain.local/v2/test/tomcat/blobs/sha256:edc06179faadbe516edba2e5b91d37e992ca0cb19c1f79a2ee2862253d0ba7b3 (15.95357ms) [body redacted: omitting binary blobs from logs]
2023/11/16 11:48:51 HTTP/1.1 200 OK
Content-Length: 247
Accept-Ranges: bytes
Cache-Control: max-age=31536000
Connection: keep-alive
Content-Security-Policy: frame-ancestors 'none'
Content-Type: application/octet-stream
Date: Thu, 16 Nov 2023 08:48:51 GMT
Docker-Content-Digest: sha256:edc06179faadbe516edba2e5b91d37e992ca0cb19c1f79a2ee2862253d0ba7b3
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:edc06179faadbe516edba2e5b91d37e992ca0cb19c1f79a2ee2862253d0ba7b3"
Server: nginx
Set-Cookie: sid=ef3169b2a54bfa33b97bd364e1b09113; Path=/; HttpOnly
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Frame-Options: DENY
X-Request-Id: a84c1888-51b8-40ed-8ee5-8193aeedc1d1

Error: no matching signatures: crypto/rsa: verification error
main.go:69: error during command execution: no matching signatures: crypto/rsa: verification error

Debug by hashivault:

root@srv:/home# cosign verify --key hashivault://cosign harbor.domain.local/kb/test/tomcat@sha256:b74dd6880fb1cd194fe23f3d2bec0cd0e89e79b609302632eb666c09c9e77709 --insecure-ignore-tlog=true --insecure-ignore-sct=true -d

WARNING: Skipping tlog verification is an insecure practice that lacks of transparency and auditability verification for the signature.
2023/11/16 11:57:11 --> GET https://harbor.domain.local/v2/
2023/11/16 11:57:11 GET /v2/ HTTP/1.1
Host: harbor.domain.local
User-Agent: cosign/v2.1.1 (linux; amd64) go-containerregistry/v0.15.2
Accept-Encoding: gzip

2023/11/16 11:57:11 <-- 401 https://harbor.domain.local/v2/ (67.551572ms)
2023/11/16 11:57:11 HTTP/1.1 401 Unauthorized
Content-Length: 76
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Thu, 16 Nov 2023 08:57:11 GMT
Docker-Distribution-Api-Version: registry/2.0
Server: nginx
Set-Cookie: sid=65118b27beaa2f1dcf71a3a98b079eed; Path=/; HttpOnly
Www-Authenticate: Bearer realm="https://harbor.domain.local/service/token",service="harbor-registry"
X-Request-Id: f23d8802-2292-4e21-b316-c28d6c98176a

{"errors":[{"code":"UNAUTHORIZED","message":"unauthorized: unauthorized"}]}

2023/11/16 11:57:11 --> GET https://harbor.domain.local/service/token?scope=repository%3Akb%2Ftest%2Ftomcat%3Apull&service=harbor-registry [body redacted: basic token response contains credentials]
2023/11/16 11:57:11 GET /service/token?scope=repository%3Akb%2Ftest%2Ftomcat%3Apull&service=harbor-registry HTTP/1.1
Host: harbor.domain.local
User-Agent: cosign/v2.1.1 (linux; amd64) go-containerregistry/v0.15.2
Authorization:
Accept-Encoding: gzip

2023/11/16 11:57:11 <-- 200 https://harbor.domain.local/service/token?scope=repository%3Akb%2Ftest%2Ftomcat%3Apull&service=harbor-registry (49.038027ms) [body redacted: basic token response contains credentials]
2023/11/16 11:57:11 HTTP/1.1 200 OK
Connection: keep-alive
Content-Security-Policy: frame-ancestors 'none'
Content-Type: application/json; charset=utf-8
Date: Thu, 16 Nov 2023 08:57:11 GMT
Server: nginx
Set-Cookie: sid=ca057432d212faa21cbc90d40900f459; Path=/; Secure; HttpOnly
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Frame-Options: DENY
X-Request-Id: 8ca1dd45-597e-45ed-b150-f4bbb25fa7de

2023/11/16 11:57:11 --> GET https://harbor.domain.local/v2/test/tomcat//referrers/sha256:b74dd6880fb1cd194fe23f3d2bec0cd0e89e79b609302632eb666c09c9e77709
2023/11/16 11:57:11 GET /v2/test/tomcat//referrers/sha256:b74dd6880fb1cd194fe23f3d2bec0cd0e89e79b609302632eb666c09c9e77709 HTTP/1.1
Host: harbor.domain.local
User-Agent: cosign/v2.1.1 (linux; amd64) go-containerregistry/v0.15.2
Accept: application/vnd.oci.image.index.v1+json
Authorization:
Accept-Encoding: gzip

2023/11/16 11:57:11 <-- 200 https://harbor.domain.local/v2/test/tomcat//referrers/sha256:b74dd6880fb1cd194fe23f3d2bec0cd0e89e79b609302632eb666c09c9e77709 (9.45321ms)
2023/11/16 11:57:11 HTTP/1.1 200 OK
Content-Length: 300
Connection: keep-alive
Content-Security-Policy: frame-ancestors 'none'
Content-Type: application/json; charset=utf-8
Date: Thu, 16 Nov 2023 08:57:11 GMT
Server: nginx
Set-Cookie: sid=ccd90f371c3b007236f79940a5bd11d1; Path=/; HttpOnly
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Frame-Options: DENY
X-Request-Id: 02ca0998-a6d4-4f45-8b6b-06958f37e87e
X-Total-Count: 1

{"schemaVersion":2,"mediaType":"application/vnd.oci.image.index.v1+json","manifests":[{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:b45b2da7cc36031dc256469e621f56dece4fbf7e2bda3559920c83af35267d69","size":1641,"artifactType":"application/vnd.oci.image.config.v1+json"}]}

2023/11/16 11:57:11 --> GET https://harbor.domain.local/v2/test/tomcat//manifests/sha256-b74dd6880fb1cd194fe23f3d2bec0cd0e89e79b609302632eb666c09c9e77709.sig
2023/11/16 11:57:11 GET /v2/test/tomcat//manifests/sha256-b74dd6880fb1cd194fe23f3d2bec0cd0e89e79b609302632eb666c09c9e77709.sig HTTP/1.1
Host: harbor.domain.local
User-Agent: cosign/v2.1.1 (linux; amd64) go-containerregistry/v0.15.2
Accept: application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json
Authorization:
Accept-Encoding: gzip

2023/11/16 11:57:11 <-- 200 https://harbor.domain.local/v2/test/tomcat//manifests/sha256-b74dd6880fb1cd194fe23f3d2bec0cd0e89e79b609302632eb666c09c9e77709.sig (15.9058ms)
2023/11/16 11:57:11 HTTP/1.1 200 OK
Content-Length: 1146
Connection: keep-alive
Content-Security-Policy: frame-ancestors 'none'
Content-Type: application/vnd.oci.image.manifest.v1+json
Date: Thu, 16 Nov 2023 08:57:11 GMT
Docker-Content-Digest: sha256:b45b2da7cc36031dc256469e621f56dece4fbf7e2bda3559920c83af35267d69
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:b45b2da7cc36031dc256469e621f56dece4fbf7e2bda3559920c83af35267d69"
Server: nginx
Set-Cookie: sid=a9a06fdfc8f5b2af12a30eadff590bc5; Path=/; HttpOnly
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Frame-Options: DENY
X-Request-Id: a94210a2-63d4-4b76-96a5-91b6b27f472e

{"schemaVersion":2,"mediaType":"application/vnd.oci.image.manifest.v1+json","config":{"mediaType":"application/vnd.oci.image.config.v1+json","size":248,"digest":"sha256:8017b161b26b330beec138cc4bd529021c76c7e8d2bc360c52bd0e90bdad9461"},"layers":[{"mediaType":"application/vnd.dev.cosign.simplesigning.v1+json","size":247,"digest":"sha256:edc06179faadbe516edba2e5b91d37e992ca0cb19c1f79a2ee2862253d0ba7b3","annotations":{"dev.cosignproject.cosign/signature":"kC6ENoScap3Du8cfjs0pjxaRlbN6hDhjn0KlPQpwdnoLwSzluD7BJUwJdRh2VV9NzqT1SJ0TThGl50iIOXkqJWdFxNJP+cCPWC46ZX6zhSRemkSBAOqjELsRGWFNV5RMkPQaG+jtdV59h033cKIkhuZGOeCJa0WNF7uc7+eBdj63TrtwC1RtP0tcWSdeUvYhTZJXamyxKFGQub3B6vL4z5QL04c7Yy6DmlyFDZLhs1Fq4c7X+a2OOTSTjaDJJuMPxxFO7EvJ+hD7uIbEDUOzjb+KLfCaKT6zZxFfIgihDHzx79zA5ob6fMwquZNXWtsBapTEhZx4DDtuvkJYw0Xz6+tnQlZ7Im8THUaWqOwu6Yjgn0pFr7LRrF3Ffcz2c63hFHFPquzOTKj2A8G4EbEXA+Avvj5ERoomFFxQvB1w+pph1D0fsmXuVjkGTm0/zEkZH73tzjQ34dbi8uRtawJxXMZ2C8A0nZPIv6WpvV/P7lJjFd6vXerZVUD5lXLTXopEu/re5W1b+k8UAkYhIfh1GPBxzsIo2HKcgc3As37GqAVfXE9Jebu0GEAr8h2mCVjg11rsoA+d6tELnajhRKcaK8jmk0csu/9xVlfLOJTryhNKgDKKZ9j3WJ41YybrKR2r/gTfiz8EDgRAnS1J2e+MAPdOgNXWVf8Un6LE3C8lbk8="}}]}
2023/11/16 11:57:11 --> GET https://harbor.domain.local/v2/test/tomcat//blobs/sha256:edc06179faadbe516edba2e5b91d37e992ca0cb19c1f79a2ee2862253d0ba7b3 [body redacted: omitting binary blobs from logs]
2023/11/16 11:57:11 GET /v2/test/tomcat//blobs/sha256:edc06179faadbe516edba2e5b91d37e992ca0cb19c1f79a2ee2862253d0ba7b3 HTTP/1.1
Host: harbor.domain.local
User-Agent: cosign/v2.1.1 (linux; amd64) go-containerregistry/v0.15.2
Authorization:
Accept-Encoding: gzip

2023/11/16 11:57:11 <-- 200 https://harbor.domain.local/v2/test/tomcat//blobs/sha256:edc06179faadbe516edba2e5b91d37e992ca0cb19c1f79a2ee2862253d0ba7b3 (17.899596ms) [body redacted: omitting binary blobs from logs]
2023/11/16 11:57:11 HTTP/1.1 200 OK
Content-Length: 247
Accept-Ranges: bytes
Cache-Control: max-age=31536000
Connection: keep-alive
Content-Security-Policy: frame-ancestors 'none'
Content-Type: application/octet-stream
Date: Thu, 16 Nov 2023 08:57:11 GMT
Docker-Content-Digest: sha256:edc06179faadbe516edba2e5b91d37e992ca0cb19c1f79a2ee2862253d0ba7b3
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:edc06179faadbe516edba2e5b91d37e992ca0cb19c1f79a2ee2862253d0ba7b3"
Server: nginx
Set-Cookie: sid=7e24ddb15e7a7164be7aac51095933dc; Path=/; HttpOnly
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Frame-Options: DENY
X-Request-Id: 1a6b5989-4836-4e3d-b09b-6874de041d1b

Verification for harbor.domain.local/test/tomcat/@sha256:b74dd6880fb1cd194fe23f3d2bec0cd0e89e79b609302632eb666c09c9e77709 --
The following checks were performed on each of these signatures:

The cosign claims were validated
The signatures were verified against the specified public key

Additional context
Policy in vault:
path "transit/keys/cosign" {
capabilities = ["read"]
}

path "transit/hmac/cosign/*" {
capabilities = ["update"]
}

path "transit/sign/cosign/*" {
capabilities = ["update"]
}

path "transit/verify/cosign" {
capabilities = ["update"]
}