Failure to refresh public keys causes IdTokenVerifier to fail valid tokens
erikjoh opened this issue · comments
After rolling out the v1.33.3 release, we've experienced a few transient 1 hour outages for OIDC token verification.
Logs show:
message: WARNING: Failed to get a certificate from certificate location https://www.googleapis.com/oauth2/v3/certs
And then all attempts to validate tokens fail with:
com.google.api.client.auth.openidconnect.IdTokenVerifier$VerificationException: Could not find PublicKey for provided keyId: 486f16482005a2cdaf26d9214018d029ca46fb56
For 1 hour straight until the cache of the empty map expires and a new cert refresh succeeds:
We believe that the implementation should not evict previously known public keys in the event of a refresh failure (which could happen for many different reasons).
Thanks for reporting. The 1.33.3 adds an important security fix that requires public keys to be fetched from the public store. It look like sometimes the store is unaccessible at the time of public key fetch and library then does not retry for an hour. We will fix that. After the fix you can expect some rate of failures is the public key store is unavailable, but the 1 hour delay will be gone.
To make it even better for the use case when the Verifier of the library already used by a wrapper with its owned signature verification - we will add a constructor that disables underlying validation. And we will update Google-owned wrappers to instantiate it correctly.