PointGFp_Multi_Point_Precompute gives wrong result when an infinity point occurs in the precomputation

[Andrew Kozlik]

We were recently contacted by @guidovranken, who alerted us that our signature verification function in https://github.com/trezor/trezor-firmware/tree/master/crypto accepts the following signature as valid whereas Botan does not:

ecc curve: secp256k1
public key X: 55066263022277343669578718895168534326250603453777594175500187360389116729240
public key Y: 83121579216557378445487899878180864668798711284981320763518679672151497189239
msg_digest: {0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11} (32 bytes)
signature R: 110618813224107091100351766566588261013518646361399424304146461958647130377927
signature S: 56528019055117870811188539769759161932852696818058491284544029456598522370972

I believe that the signature is valid and the bug is in Botan's PointGFp_Multi_Point_Precompute class. What's special about the given example is that the public key is -G (the negation of the base point). So when you use PointGFp_Multi_Point_Precompute in the ECDSA_Verification_Operation class, the input to its constructor is G and -G. Thus the precomputation of the 15 values for Shamir's trick results in several infinity points. However, the class does not take this possibility into account because it's written to work with the affine point representation. When you call PointGFp::force_all_affine() on these points in the constructor, the function finishes without throwing any exceptions, but all the precomputed points are now zeroed. As a result, when you compute the value of R in ECDSA_Verification_Operation::verify(), it comes out to be the infinity point and signature validation inevitably fails.

At the very least this scenario should result in an exception being thrown by PointGFp::force_all_affine(), just like PointGFp::force_affine() does. But to be fair, -G is a valid public key, so PointGFp_Multi_Point_Precompute should be fixed to accommodate this scenario.

[Guido Vranken]

Thank you for researching this in-depth @andrewkozlik.

[Jack Lloyd]

I believe you are correct this is a bug in Botan. Thank you @andrewkozlik for the report and analysis, and as always @guidovranken thanks for cryptofuzz