OpenSSL module: avoidable large allocations in OpKDF targets

[Santiago Zanella-Beguelin]

OpKDF targets use malloc to allocate arbitrarily large chunks of memory upfront to store the output of EVP_PKEY_derive, but EVP_PKEY_derive may fail without every using that memory because the requested output length exceeds the maximum allowable length.

See for instance https://github.com/guidovranken/cryptofuzz/blob/master/modules/openssl/module.cpp#L1871

These large allocations could be avoided by first calling EVP_PKEY_derive to determine the maximum output length and checking that the length requested is within bounds (https://www.openssl.org/docs/man1.1.0/man3/EVP_PKEY_derive.html). See s-zanella@7699bb6 for a way of doing this.

I'm not sure this is an issue in practice because the memory requested to malloc may never materialize. Doing these checks before calling EVP_PKEY_derive means that the logic to check bounds would not be exercised in tests (however, the logic to get the maximum output length will be).