ECDSA verification compatibility issue due to applying different hash truncation
guidovranken opened this issue · comments
Botan uses a slightly different ECDSA input truncation mechanism than OpenSSL/LibreSSL/BoringSSL/wolfCrypt.
https://nvlpubs.nist.gov/nistpubs/fips/nist.fips.186-4.pdf says:
When the length of the output of the hash function is greater than the bit length of n, then the leftmost n bits of the hash function output block shall be used in any calculation using the hash function output during the generation or verification of a digital signature.
This is what Botan does.
OpenSSL and the others do this: https://github.com/openssl/openssl/blob/5357c10624bedaeed984ef4ff370096911ee2ddf/crypto/ec/ecdsa_ossl.c#L245-L258
So I think Botan is actually right and the others are doing it slightly wrong.
In my fuzzer, I apply the same truncation mechanism before calling Botan's verify_message
: https://github.com/guidovranken/cryptofuzz/blob/7f9f35e20fcd8886cbcbf167474dd7f9954e5c02/modules/botan/module.cpp#L726-L739
So for me it's not a problem, but in practice this can lead to interoperability issues (Botan rejecting valid signatures).
Here's an example of a valid signature that Botan rejects:
operation name: ECDSA_Verify
ecc curve: x962_p239v1
public key X: 115014969141122710336858256331515905188079709631033705848897690363377891
public key Y: 81333916963110019576228330948951168219884247801435258672405011123948094
cleartext: {0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00} (30 bytes)
signature R: 336870737920796383793526100561731463195539556177859437212670821204992455
signature S: 328556355548848640685841521416759180891403359973782869535609514452377183
Module Botan result:
false
Module OpenSSL result:
true
(numbers are in decimal, cleartext must be passed through SHA256 first)
Interesting bug as we attempted to match OpenSSL's behavior in #1502 but I think the logic is slightly wrong, specifically if the hash has leading zero bits. Somehow the existing test cases missed this. Thanks for the report!
Further twist - if I change the logic, several RFC 6979 vectors fail to validate. I wonder how these other implementations handle these signatures.