BN_mod_inverse incorrect result when parameters are aliased

[Guido Vranken]

#include <openssl/bn.h>

#define CF_CHECK_NE(expr, res) if ( (expr) == (res) ) { goto end; }

int main(void)
{
    char* str = NULL;
    BIGNUM* a = BN_new();
    BIGNUM* b = BN_new();
    BN_CTX* ctx = BN_CTX_new();
    CF_CHECK_NE(BN_dec2bn(&a, "5193817943"), 0);
    CF_CHECK_NE(BN_dec2bn(&b, "3259122431"), 0);
    CF_CHECK_NE(BN_mod_inverse(b, a, b, ctx), NULL);
    str = BN_bn2dec(b);
    printf("%s\n", str);
end:
    BN_free(a);
    BN_free(b);
    BN_CTX_free(ctx);
    OPENSSL_free(str);
    return 0;
}

This prints 0 but should print 2609653924. Note that in the call to BN_mod_inverse, the result is the same pointer as the modulus. The documentation explicitly states that "r may be the same BIGNUM as a or n.".

This also affects @davidben and @botovq.

[Pauli]

It's a bug. Is it a documentation or code issue is the question I'd like answered -- I'm willing to go either way here.

[Theo Buehler]

The code obviously tries to deal with either of the two inputs being equal to the output. So it's a programming error, not a documentation error. BN_nnmod(x, x, y, ctx) works and the libraries rely on that in various places. Here we take the path where BN_nnmod(R, Y, n, ctx) with R == n. This doesn't work. The first BN_mod() call clobbers the modulus with its result and if the result happens to be negative, BN_nnmod() will return 0.

[Pauli]

That this particular code path doesn't work and probably never has is indicative that it's never used by anyone -- hence a documentation only fix is an option. As mentioned, willing to swing either way here.

[Guido Vranken]

There are also functions where aliasing is not explicitly allowed and the function will silently produce the incorrect result instead of failing.

#include <openssl/bn.h>

#define CF_CHECK_EQ(expr, res) if ( (expr) != (res) ) { goto end; }
#define CF_CHECK_NE(expr, res) if ( (expr) == (res) ) { goto end; }

int main(void)
{
    char* str = NULL;
    BIGNUM* a = BN_new();
    BIGNUM* b = BN_new();
    BIGNUM* c = BN_new();
    BN_CTX* ctx = BN_CTX_new();
    CF_CHECK_NE(BN_dec2bn(&a, "3"), 0);
    CF_CHECK_NE(BN_dec2bn(&b, "1"), 0);
    CF_CHECK_NE(BN_dec2bn(&c, "11"), 0);
    CF_CHECK_EQ(BN_mod_exp_simple(c, a, b, c, ctx), 1);
    str = BN_bn2dec(c);
    printf("%s\n", str);
end:
    BN_free(a);
    BN_free(b);
    BN_free(c);
    BN_CTX_free(ctx);
    OPENSSL_free(str);
    return 0;
}

Personally I think it's better to return an error to prevent accidental misuse.

Would each of the maintainers like to be informed about each of these?

[Theo Buehler]

Would each of the maintainers like to be informed about each of these?

I mean, I understand why I want a = a + b (mod m) to work. I can also imagine that a = a^-1 (mod m) may be useful. However, your original test is m = a^-1 (mod m). Now you tell us m = a^b (mod m) is broken. Is that ever useful? I can't come up with situation where I would want to do that and I can't make sense of this mathematically. I'm inclined to say *shrug*.

[Guido Vranken]

I don't see why variable reuse would not be useful. Programming languages have arithmetic assignment operators for this reason. But if you want to base your code on implicit assumptions about the semantics of your users be my guest.

[Theo Buehler]

Maybe I expressed myself poorly. I think that taking the modulus as output of a calculation (mod m) is strange and I would not really expect that to work. I agree with you that erroring would be much more appropriate than silently returning nonsense. So yes, it would be preferable if this could be fixed. However, I would much prefer to prioritize variable reuse of any variables other than the modulus. Once we know that these all work, I'm perfectly happy to go through the other cases to make computations more resilient.

[Theo Buehler]

Here is my submission for the original bug report to BoringSSL: https://boringssl-review.googlesource.com/c/boringssl/+/60365 and I will land a similar diff in LibreSSL soon. I haven't tested it but I'm rather sure that OpenSSL's implementation can be fixed analogously.

[David Benjamin]

Aliasing is definitely a bit of a curse. 😞 It's very useful to act on values in-place, but also makes reasoning about abstraction boundaries very hard. And while aliasing can reduce copies, having to defensively allow aliasing across API boundaries can also increase copies, which defeats the purpose of believing in aliasing in the first place.

The Rust answer would be to just forbid this kind of aliasing altogether (what the borrow checker does) and, if you want operations that need to work in-place, you make separate in-place and non-aliasing functions. (E.g. one function for a = b + c, no aliasing, and one for a = a + b where a is a single &mut input/output parameter.)

But this isn't Rust and many BIGNUM functions expect to be called with aliasing pointers already, so we're kinda stuck at least sometimes allowing aliased BIGNUMs. Agreed with @botovq that aliasing on the modulus is questionably useful. And, the way a lot of these functions work, allowing it often gets you in the "defensively allowing aliasing causes more copies than before" bucket.

Regardless, BN_mod_inverse is pretty easy to fix, so let's do that, and we ought to define and write down aliasing expectations more clearly.

[Pauli]

Would each of the maintainers like to be informed about each of these?

If you have such a list, then I think it would be worthwhile. As you note, aliasing is bad -- it should either be fixed, return an error or minimally be documented. This is one thing FORTRAN got right 😁

edit: restrict declarations would be another option I guess.

[Guido Vranken]

If you have such a list, then I think it would be worthwhile. As you note, aliasing is bad -- it should either be fixed, return an error or minimally be documented. This is one thing FORTRAN got right grin

The only ones I've found so far are BN_mod_exp_simple and BN_mod_sub_quick

[Pauli]

@bernd-edlinger are those commits part of a PR?

[Bernd Edlinger]

Yes, I've added them to #21124 as separate commits.

[Theo Buehler]

Please reconsider whether the approach taken in @bernd-edlinger's PR is really the way you want to go. Not only does that PR not fix aliasing of result and modulus in at least half the public functions in bn_mod.c that it doesn't touch (and hence it doesn't fix the issues discussed here), it also adds a considerable amount of complexity for a dubious feature that has never worked in the last quarter century. Echoing the sentiment expressed by @pauli in the initial response, I think fixing the documentation and throwing an error is an option and should be seriously considered as a much cleaner and simpler alternative. The reason I fixed BN_mod_inverse() the way I did in the other projects is not that I missed that I could have changed BN_nnmod(). I did it this way because it fixes the bug, makes things simpler and at the same time it allows me to throw an error on aliasing result and modulus in most affected functions.

[Tomáš Mráz]

I agree with @botovq that allowing aliasing the modulus in these operations does not make much sense and we should simply return error.

[Guido Vranken]

The only ones I've found so far are BN_mod_exp_simple and BN_mod_sub_quick

I've also observed errors with BN_div_recp if the result is aliased. This is a public function in OpenSSL but private in LibreSSL and BoringSSL.

[Guido Vranken]

Can the fix for this be merged please @bernd-edlinger ?

[Tomáš Mráz]

@guidovranken alternative fix in #22421 done as OTC requested - aliasing of modulus with the result should not be supported.

[Bernd Edlinger]

Now we have different results with BN_mod_exp_simple vs. BN_mod_exp_recp when aliasing result with modulus.
BN_mod_exp_recp fails and BN_mod_exp_simple succeeds - that is nonsense.