ECDSA verification compatibility issue due to applying different hash truncation

[Guido Vranken]

Botan uses a slightly different ECDSA input truncation mechanism than OpenSSL/LibreSSL/BoringSSL/wolfCrypt.

https://nvlpubs.nist.gov/nistpubs/fips/nist.fips.186-4.pdf says:

When the length of the output of the hash function is greater than the bit length of n, then the leftmost n bits of the hash function output block shall be used in any calculation using the hash function output during the generation or verification of a digital signature.

This is what Botan does.

OpenSSL and the others do this: https://github.com/openssl/openssl/blob/5357c10624bedaeed984ef4ff370096911ee2ddf/crypto/ec/ecdsa_ossl.c#L245-L258

So I think Botan is actually right and the others are doing it slightly wrong.

In my fuzzer, I apply the same truncation mechanism before calling Botan's verify_message: https://github.com/guidovranken/cryptofuzz/blob/7f9f35e20fcd8886cbcbf167474dd7f9954e5c02/modules/botan/module.cpp#L726-L739

So for me it's not a problem, but in practice this can lead to interoperability issues (Botan rejecting valid signatures).

Here's an example of a valid signature that Botan rejects:

operation name: ECDSA_Verify
ecc curve: x962_p239v1
public key X: 115014969141122710336858256331515905188079709631033705848897690363377891
public key Y: 81333916963110019576228330948951168219884247801435258672405011123948094
cleartext: {0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00} (30 bytes)
signature R: 336870737920796383793526100561731463195539556177859437212670821204992455
signature S: 328556355548848640685841521416759180891403359973782869535609514452377183

Module Botan result:

false

Module OpenSSL result:

true

(numbers are in decimal, cleartext must be passed through SHA256 first)

[Jack Lloyd]

Interesting bug as we attempted to match OpenSSL's behavior in #1502 but I think the logic is slightly wrong, specifically if the hash has leading zero bits. Somehow the existing test cases missed this. Thanks for the report!

[Jack Lloyd]

Further twist - if I change the logic, several RFC 6979 vectors fail to validate. I wonder how these other implementations handle these signatures.