spec: signing secrets, encoding, `wh*_` prefix, key discovery, questionable cryptography recommendations
hf opened this issue · comments
Encoding Bytes
In the spec so far, under the signature schemes it says:
base64 encoded, prefixed with
whsec_for easy identification.
base64 encoded, prefixed with
whsk_for the secret key, andwhpk_for the public key for easy identification.
Why was this scheme chosen, and what do the wh*_ prefixes mean?
Allow me to suggest some changes to this.
The spec says "base64 encoded" but there's actually two competing Base64 encodings in use today: Base64 standard (with = padding and non-URL-safe characters) and Base64-URL (without padding and with URL-safe characters). In some environments, like NodeJS, you can decode from any Base64 variant, and in others (Go, C, Rust) you have to specify it exactly.
I would suggest a change to the spec to specifically state Base64 or Base64-URL unambiguously.
Encoding the Secret
Why was the wh*_ prefix chosen and what does it mean? I can't think of a mnemonic for it.
In general though, I would suggest avoiding the re-implementation and re-specification of cryptographic primitives. JWT (and the associated JOSE and other RFCs) went down this route and ended up reimplementing ASN.1 messages that encoded cryptographic information. Libraries are still struggling with implementing these well, so it's my opinion that this spec should intentionally avoid reinventing the wheel from the get-go on this.
I would suggest we choose one or more accepted standards for encoding cryptographic keys and use those.
Personally I'm in favor of JWKS as it has matured sufficiently as a standard and library coverage is wide enough. Being JSON, makes it very easy to read as a human (unlike DER / ASN.1).
The singular availability of Ed25519 as an asymmetric algorithm needlessly limits the choices of asymmetric algorithms that can be used. RSA is still pretty commonly used. Use of hardware devices may not support EdDSA algorithms at all, but may be required by parties implementing the spec.
In short, let's defer to already existing and well-supported standards on encoding cryptographic key material so that we don't end up reinventing the wheel.
Discovery
One thing that OIDC did well with its JWTs is to specify a discovery mechanism for public keys, which completely decoupled the need to share secrets and other cryptographic material between "publisher" and "verifier."
Allow me to suggest the same. Instead of imagining yet-another versioning scheme about signatures, how they're encoded, and what key is being used, we could define the following:
webhook-jwksA header that is a HTTPS URL where the verifier can fetch all asymmetric signing keys for the signature. If missing the publisher and verifier should have agreed upon cryptographic material out-of-band.webhook-key-idA header containing the unique key identifier that producedwebhook-signature, which should reside withinwebhook-jwks. Must be present ifwebhook-jwksis present, optional otherwise.
This would allow for:
- Simple use-cases where publisher and verifier agree on a pre-shared key (identified with
webhook-key-idso that we don't re-imagine a new spec). - More complex use cases where the burden of the verifier having obtained the proper and currently trusted public key material is shifted to the publisher by using a HTTPS endpoint to publish its key material.
This would make it very easy to build libraries and flows with minimal ceremonies about key maintenance and scheduling.
In the same vein, allow me to finally re-introduce the JWT aud claim as the webhook-audience header. When the publisher of the webhook is a larger service with many customers, relying just on public keys can become difficult. Verifiers shouldn't just rely on cryptographic signatures to figure out if a webhook is meant for them; they should also check webhook-audience and identify if the payload was meant for them. This effectively solves most SSRF cases as well, since the receiver should reject payloads that they're not listening (hence audience) for.
Questionable Cryptographic Recommendations
I saw these considerations and wanted to challenge them:
Signing keys should be unique per endpoint for symmetric signatures, and unique per endpoint (or potentially customer) for asymmetric signatures. Reusing keys across customers can lead to security issues!
It's almost always recommended to choose symmetric signatures when possible, as the performance implications of asymmetric signatures are quite severe.
Specifically:
Reusing keys across customers can lead to security issues!
This is true if there's no additional mechanism such as my proposal for webhook-audience above. Relying just on cryptography for identification is not a great idea; this is why websites today can use the same public key in their TLS cert, but advertise a different CN.
It often is much easier, security wise, to have one set of private keys residing in a hardware security module for the whole platform (that publishes webhooks) rather than maintain many signing keys per customer / endpoint. Hardware modules are not cheap (nor are services like AWS KMS), so it's cost-prohibitive to maintain one-per-customer of those. Some platforms may be required to use hardware modules (especially FIPS compliant ones) by compliance or law for any cryptography.
It's almost always recommended to choose symmetric signatures when possible, as the performance implications of asymmetric signatures are quite severe.
In fact, our recommendation should be the exact opposite. Asymmetric signatures are marginally slower (compute-wise) than symmetric signatures. Performance is a non-goal in this space, as having good security practices is more important than shaving sub-microsecond performance issues.
By all accounts, it's much worse if anyone -- even for experiments -- uses symmetric keys. Yes, they're useful in some cases, but our recommendation should always be for people to use asymmetric keys. It's much easier to rotate them when/if they leak (going back to my webhooks-jwks recommendation!), no secrets need to be shared, only public information (so only one system component needs to know the private key), etc.
In general I would strive to avoid any unfounded (who recommends and where?) or superfluous security recommendations in hopes of not leading library implementers and developers astray from the main security goals.
Hey, sorry for forgetting to comment on it until now.
Encoding Bytes
In the spec so far, under the signature schemes it says:
base64 encoded, prefixed with
whsec_for easy identification.base64 encoded, prefixed with
whsk_for the secret key, andwhpk_for the public key for easy identification.Why was this scheme chosen, and what do the
wh*_prefixes mean?
Stands for "webhook".
As for why was it chosen? Two reasons:
- Secret scanners - have a way to tag secrets so that they can be flagged by secret scanners (e.g. if someone commits them to git).
- It makes it harder to accidentally use a token or the wrong kind of identifier in the wrong place. It's helpful both when debugging issues and you can even add this checking in code.
Allow me to suggest some changes to this.
The spec says "base64 encoded" but there's actually two competing Base64 encodings in use today: Base64 standard (with
=padding and non-URL-safe characters) and Base64-URL (without padding and with URL-safe characters). In some environments, like NodeJS, you can decode from any Base64 variant, and in others (Go, C, Rust) you have to specify it exactly.I would suggest a change to the spec to specifically state Base64 or Base64-URL unambiguously.
The intention was standard base64, not the URL variant. Feel free to open a PR to clarify the wording there!
Encoding the Secret
Why was the
wh*_prefix chosen and what does it mean? I can't think of a mnemonic for it.
As said above: webhook.
In general though, I would suggest avoiding the re-implementation and re-specification of cryptographic primitives. JWT (and the associated JOSE and other RFCs) went down this route and ended up reimplementing ASN.1 messages that encoded cryptographic information. Libraries are still struggling with implementing these well, so it's my opinion that this spec should intentionally avoid reinventing the wheel from the get-go on this.
I would suggest we choose one or more accepted standards for encoding cryptographic keys and use those.
Personally I'm in favor of JWKS as it has matured sufficiently as a standard and library coverage is wide enough. Being JSON, makes it very easy to read as a human (unlike DER / ASN.1).
The singular availability of Ed25519 as an asymmetric algorithm needlessly limits the choices of asymmetric algorithms that can be used. RSA is still pretty commonly used. Use of hardware devices may not support EdDSA algorithms at all, but may be required by parties implementing the spec.
In short, let's defer to already existing and well-supported standards on encoding cryptographic key material so that we don't end up reinventing the wheel.
I completely agree with the part about ED25519. It's also not FIPS-140 and we should probably address that. I also think we should probably allow for a couple of other alternatives and version them better. The reason to go with DSA rather than RSA is that RSA is very slow to sign and fast to verify. DSA is the opposite. We want the webhook sender (which is probably sending way more than each receiver is receiving) not to be so slow.
As for the use of JWKS: the spec in general (and this part in particular) were mostly done to follow and cement best practices, rather than coming up with new best practices. This was to make it easier for people to adopt it. In this particular case, I also prefer the best practice rather than relying on JWKS/JWT/JWS. Sometimes not having configurability or flexibility, and just having to use best practices is a good thing. Look for example at Wireguard vs. OpenVPN. I attribute some of the success of the former in usurping the latter lies in the simplicity. There's only one cipher (well, one asymmetric and one optional symmetric), and almost no configuration for the security side.
I think we should definitely support asymmetric (maybe even Ed25519 as well as a FIPS curve, or just a FIPS curve), but we should just define it as a version. E.g. at the moment we have v1 for symmetric and v1a for Ed25519. We can potentially make the versioning nicer, but having it be a defined version of the spec that defines the cipher suite, and if it's ever not enough, we can change to a new one feels simpler/better in my pov. Significantly decreases the chances of incompatibilities between client and server.
Discovery
One thing that OIDC did well with its JWTs is to specify a discovery mechanism for public keys, which completely decoupled the need to share secrets and other cryptographic material between "publisher" and "verifier."
Allow me to suggest the same. Instead of imagining yet-another versioning scheme about signatures, how they're encoded, and what key is being used, we could define the following:
* `webhook-jwks` A header that is a HTTPS URL where the verifier can fetch all asymmetric signing keys for the signature. If missing the publisher and verifier should have agreed upon cryptographic material out-of-band. * `webhook-key-id` A header containing the unique key identifier that produced `webhook-signature`, which should reside within `webhook-jwks`. Must be present if `webhook-jwks` is present, optional otherwise.This would allow for:
* Simple use-cases where publisher and verifier agree on a pre-shared key (identified with `webhook-key-id` so that we don't re-imagine a new spec). * More complex use cases where the burden of the verifier having obtained the proper and currently trusted public key material is shifted to the publisher by using a HTTPS endpoint to publish its key material.This would make it very easy to build libraries and flows with minimal ceremonies about key maintenance and scheduling.
In the same vein, allow me to finally re-introduce the JWT
audclaim as thewebhook-audienceheader. When the publisher of the webhook is a larger service with many customers, relying just on public keys can become difficult. Verifiers shouldn't just rely on cryptographic signatures to figure out if a webhook is meant for them; they should also checkwebhook-audienceand identify if the payload was meant for them. This effectively solves most SSRF cases as well, since the receiver should reject payloads that they're not listening (hence audience) for.
I need to think about it a bit more to really have a strong opinion one way or the other. Though I think this one also falls into the "meet people where they are" category. As in, much better to have a spec that's imperfect that everyone adopts, than one that's perfect but no one does. Doing something that's already very close to industry best practices, e.g. current version is similar to what's done at Stripe, Mux, Svix, Github, Shopify, Zoom, etc. so I think adhering to that is good.
Questionable Cryptographic Recommendations
I saw these considerations and wanted to challenge them:
Additional considerations:
Signing keys should be unique per endpoint for symmetric signatures, and unique per endpoint (or potentially customer) for asymmetric signatures. Reusing keys across customers can lead to security issues!
It's almost always recommended to choose symmetric signatures when possible, as the performance implications of asymmetric signatures are quite severe.Specifically:
Reusing keys across customers can lead to security issues!
This is true if there's no additional mechanism such as my proposal for
webhook-audienceabove. Relying just on cryptography for identification is not a great idea; this is why websites today can use the same public key in their TLS cert, but advertise a different CN.It often is much easier, security wise, to have one set of private keys residing in a hardware security module for the whole platform (that publishes webhooks) rather than maintain many signing keys per customer / endpoint. Hardware modules are not cheap (nor are services like AWS KMS), so it's cost-prohibitive to maintain one-per-customer of those. Some platforms may be required to use hardware modules (especially FIPS compliant ones) by compliance or law for any cryptography.
It's almost always recommended to choose symmetric signatures when possible, as the performance implications of asymmetric signatures are quite severe.
In fact, our recommendation should be the exact opposite. Asymmetric signatures are marginally slower (compute-wise) than symmetric signatures. Performance is a non-goal in this space, as having good security practices is more important than shaving sub-microsecond performance issues.
By all accounts, it's much worse if anyone -- even for experiments -- uses symmetric keys. Yes, they're useful in some cases, but our recommendation should always be for people to use asymmetric keys. It's much easier to rotate them when/if they leak (going back to my
webhooks-jwksrecommendation!), no secrets need to be shared, only public information (so only one system component needs to know the private key), etc.In general I would strive to avoid any unfounded (who recommends and where?) or superfluous security recommendations in hopes of not leading library implementers and developers astray from the main security goals.
Regarding performance, it's not negligible, at least when looking at RSA signing, though I do agree with the sentiment.
This is the performance on my laptop (used openssl speed for benchmarks) for signing/verify:
SHA256: ~10m / s (for both, it's symmetric)
Ed25519: sign: 29k / s, verify: 11k / s
RSA2048: sign: 4.4k / s, verify: 62k / s.
It'll be worse on a slower machine, especially one that's doing other things (e.g. actually sending the webhooks). Verifying Ed25519 can also become significant for a server an API gateway (e.g. Kong) having to verify high loads of webhooks. Though this is potentially premature optimization - it's just that with a spec, you can only prematurely optimize, never after the fact, so you want to be extra careful there.
With that being said I agree with the sentiment, especially about having one the key being stored in hardware. Though there suggested solution requires an additional step to be secure. One security issue with sharing the key across endpoints is that people can do replay attacks on the request. So consider for example the sender Alice and receiver Bob. Eve then gets Alice to send it a test webhook that's signed with the correct key. Even can then send that to Bob, and the webhook will verify correctly (as it was signed correctly).
To solve against it, you need to sign a unique per-endpoint token, e.g. the URL (though that's a terrible one because of canonization, proxies, pain of configuration, etc) which the verifier needs to then verify to ensure is secure. It's therefore much better to have a random key that's unique per endpoint that's also signed. Though at that point you haven't really simplified key sharing at all, as the endpoint would need to verify it's the correct key.
That's why per-endpoint keys are so nice, they are dead simple. Though let me know if you have any ideas on how to make this simpler to use, I'm sure there is a way! I think even just what I described above with the extra token is probably better than what's already there in the spec because you have the benefit of being able to put the key in the HSM.
Looking forward to hearing your thoughts, and apologies again for the delay!
@hf, any thoughts?
Yeah I generally agree this can be moved to future versions of the spec. I'd still like to not jump to conclusions about "speed." I'd rather have people choose Ed25519 over HMAC any day of the week. Most software that does any TLS already does signing, ECDH and all that overhead so one more signature is absolutely negligible in the grand scheme of things. Nothing happens over HTTP anyway.
Yeah I generally agree this can be moved to future versions of the spec. I'd still like to not jump to conclusions about "speed." I'd rather have people choose Ed25519 over HMAC any day of the week. Most software that does any TLS already does signing, ECDH and all that overhead so one more signature is absolutely negligible in the grand scheme of things. Nothing happens over HTTP anyway.
Yes, I fully agree. The thinking wasn't about the signing (which is fast), it was about the verification for people receiving many webhooks a second. Though to be honest, this is not different to people making many API calls (where they need to verify the server).
So in short, I fully agree. I think the next step is to remove asymmetric + the recommendation to do symmetric, and then open a PR adding it back based on your feedback above. Happy to do that and get it for you to take a look.
If you want to, do it. Otherwise I'll have some cycles on the weekend to do it -- kinda busy weeks at Supabase.
Ha, I'm sure with launch week coming up. I think I should be able to carve out time for it before the weekend. If not super happy for you to do it. If I end up doing it, you'll need to review/comment on my PR anyway, so make sure you at least have cycles for that.
I opened #28 as a first step. Either of us can do the second part (adding it back) once we have time (I'll let you know if I find some), but at least the spec can be cleaned of the misleading recommendations.
@hf, do you think you will manage to get to it, or need me to take a stab at it?
Everything in this ticket has now been addressed and the PR about asymmetric signatures has been merged, so I'm closing it. Please reopen it in case I missed something.
Additional discussion about asymmetric signatures is now taking place at #34.