Pluggable Signatures mechanism implementation approaches
alexmikhalevich opened this issue · comments
A couple of weeks ago, you mentioned that OpenSSL project could be interested in implementing a Pluggable Signatures mechanism. Such a mechanism potentially allows users to use custom signature schemes (e.g., post-quantum) in TLS without libssl modification. As we mentioned, we are willing to take part in this activity. In this issue, we would like to propose two possible Pluggable Signatures mechanism implementation approaches and to get some feedback from you.
Allowing custom sigalgs
In this implementation approach, we propose to use not only built-in sigalgs, but also to add a custom sigalgs list. Signatures from such a list should be treated as supported and allowed in TLS. We also assume a simple API to add/remove elements of this list here.
TLS extension
In this implementation approach, we propose to implement a TLS extension specification. Such an extension could keep an additional sigalgs or a list of sigalgs to be checked. For instance, we could use something similar to the Extension For Establishing An Additional Shared Secret. We propose to treat the aforementioned extension as built-in (its definition should be present in ext_defs (link)). However, we think it would be better to hide it behind the enable_custom_sigalgs runtime option. We propose to pass a list of sigalgs IDs to be checked via the new OpenSSL configuration option (in the configuration file).
In both proposals, we assume the development and integration of an API to map additional sigalgs NIDs to the corresponding TLS IDs (see SignatureScheme, link).
Among the benefits of the first approach, we would like to emphasize its simplicity and the absence of the necessity of TLS state machine modification. Also, it does not contradict the TLS standard. On the other hand, the second approach seems more flexible for us. In this case, we can check multiple sigalgs instead of one as in the previous method. To our point of view, this is the decisive advantage, even considering the contradiction to the TLS specification.
We would be grateful to hear your opinion on the described approaches.
If I understood what you wrote correctly in your second option you are proposing to add a non-standard extension in order to achieve this? This is not going to be an acceptable way forward. As a policy we only implement standards based solutions.
A pluggable signature scheme needs to work in standards compliant settings and not just for the purposes of research. I know that your particular interest is researching post-quantum signature schemes. However I see applications for this outside of research labs - for example to plug in support for whatever National crypto algorithms are mandated for any particular region.
I would expect to see a solution based on a custom sigalgs list. However I don't see this as an application facing API but rather a provider facing API (see provider docs here). Note that providers are a new concept being introduced into OpenSSL 3.0 and are a work-in-progress.
We already have the ability to plug in libcrypto level signature schemes via the OSSL_OP_SIGNATURE operation (see docs here). So I see two possible viable approaches:
- Extend the existing OSSL_OP_SIGNATURE operation to enable some TLS specific mode via parameters, and add the ability to "get" the TLS SignatureScheme id from the implementation. libssl could then query all available providers for OSSL_OP_SIGNATURE implementations with the "tls" property set to "yes". For any that it finds it queries the implementation for its SignatureScheme id and adds it to its internal list.
or
- Introduce a new operation (e.g. OSSL_OP_TLS_SIGNATURE). This operation would be specifically targeted at TLS and provide functions specifically for that purpose. libssl would simply query all available providers for implementations of this operation and ask each implementation for the SignatureScheme id to be used.
Of these two options I tend towards the latter.
The advantage of this provider based approach is that it will work with standard applications, and with a standard build of OpenSSL without any recompilation necessary to use the new signature scheme. To enable an application to use the new signature scheme you just build the provider and put it in the system provider directory and modify config files to ensure the new provider is loaded.
First of all, thank you for your answer.
Yes, you've got our second option correctly. I want to emphasize that despite this option contradicts the standard, such a solution is still compliable with it.
In addition, could you please clarify what particular cases of national crypto algorithms do you have in mind?
In addition, could you please clarify what particular cases of national crypto algorithms do you have in mind?
As one of the implementors of the national crypto, I can provide such a case — GOST algorithms. Currently, you can find GOST stuff looking for #ifndef OPENSSL_NO_GOST in the ssl/ folder. Support of this specification https://tools.ietf.org/html/draft-smyshlyaev-tls12-gost-suites-06 is currently WIP.
I want to emphasize that despite this option contradicts the standard, such a solution is still compliable with it.
Nonetheless this is not an option that the project could accept.
In addition, could you please clarify what particular cases of national crypto algorithms do you have in mind?
Well @beldmit has already spoken up around GOST. @InfoHunter has also expressed interest in this feature wrt the Chinese ciphers. I imagine there may be other similar use cases we haven't thought of.
BTW another thought came to mind around this feature. We will have to come up with some solution for adding unknown certificate types. At the moment, in a server, we can support exactly 1 RSA, certificate, 1 DSA certificate, 1 ECDSA certificate etc. There are a number of certificate "slots" - one for each certificate type. We're going to have to change that logic because there is no point in having the capability to have pluggable signature schemes if we can't support certificates containing public keys that match those signature schemes.
Something else to consider is what protocol version we wish to support. Making all this work in TLSv1.3 is simpler. However in TLSv1.2 and below the ciphersuites also define the authentication algorithm to use (so there is some overlap with the signature algorithm extension). If we wish to support TLSv1.2 then we may also need to have pluggable ciphersuites.
I know that your particular interest is researching post-quantum signature schemes. However I see applications for this outside of research labs - for example to plug in support for whatever National crypto algorithms are mandated for any particular region.
Currently, we are thinking not only in terms of research but in terms of delivering our post-quantum signatures and key exchange algorithms to clients. We are ready to contribute to the OpenSSL project to provide our clients with new exciting features.
Considering our needs and community needs, we are looking for some trade-off option to integrate PQC into the most widely used cryptographic runtimes.
We convinced that the mechanism we are discussing could be useful for different purposes, e.g.:
- research people could try new algorithms in the real environment more easily and conveniently;
- implementors of the national crypto, which should stick to the national standards, could integrate the related stuff more naturally.
Moreover, we have been noticing some amount of such requests already. As far as we understand, your goal here is to reduce support costs for such requests till the v3.0 release and avoid impact on your codebase. However, according to your release schedule, OpenSSL v1.1.1 is going to be supported and used at least until 2023. So we guess it seems to be reasonable to implement such an approach for 1.1.1 branch too, for TLS v1.3 only though (it makes no sense to implement the early adopters feature in the legacy standard).
The advantage of this provider based approach is that it will work with standard applications, and with a standard build of OpenSSL without any recompilation necessary to use the new signature scheme.
We are targeting the same goal, however, thinking such a feature could be useful in the context of the v1.1.1 build.
Did we get correctly that in the provider-based approach, you have in mind adding non-standard signature identifiers into the standard TLS v1.3 packet field?
libssl would simply query all available providers for implementations of this operation and ask each implementation for the SignatureScheme id to be used.
It probably could happen that different communication sides have different TLS implementation (customized OpenSSL, GnuTLS + OpenSSL, etc.). In such a case, there may be different algorithms defined with the same SignatureScheme ID. Could such a problem take place, and if it does - how could it be prevented?
Also, do we understand correctly that by certificate slots you mean a place in the certificate request packet where an unknown certificate type should be stored? Or you have in mind only available certificate types list extension in the inner OpenSSL structures?
We are targeting the same goal, however, thinking such a feature could be useful in the context of the v1.1.1 build.
This is not possible. We do not add new features to stable releases - only bug/security fixes. This feature can only be added to v3.0 at the earliest.
Did we get correctly that in the provider-based approach, you have in mind adding non-standard signature identifiers into the standard TLS v1.3 packet field?
I have in mind the ability for providers to specify the signature identifier rather than having them built-in, i.e. so that OpenSSL itself does not need to know them in advance. I see this being used in one of two ways:
- In "reasearch only" or "closed" environments (i.e. not general internet) providers might use one of the signature scheme identifiers that are reserved for private use (i.e. identifiers in the range 0xFE00-0xFFFF)
or
- Where the intention is to use a signature scheme on the open internet then provider authors can register their own signature schemes with IETF and have them used by OpenSSL immediately without having to wait for a new version with support built-in.
I have no intention of supporting non standard identifiers on the internet (people like @richsalz and @kaduk would most likely be very upset with me if I proposed that!).
It probably could happen that different communication sides have different TLS implementation (customized OpenSSL, GnuTLS + OpenSSL, etc.). In such a case, there may be different algorithms defined with the same SignatureScheme ID. Could such a problem take place, and if it does - how could it be prevented?
In the two scenarios I outlined above this should not happen. Either you're in a closed environment in which case you have control over what signature schemes are being used. Or you're on the internet in which case your signature scheme has a registered id.
Also, do we understand correctly that by certificate slots you mean a place in the certificate request packet where an unknown certificate type should be stored? Or you have in mind only available certificate types list extension in the inner OpenSSL structures?
I am talking about the certificates supported in the internal OpenSSL structures. More specifically:
Lines 1897 to 1913 in 3c957bc
In the definition of the cert_st structure we have an array of CERT_PKEYs called "pkeys". This has SSL_PKEY_NUM elements in it. Each integer from 0 to SSL_PKEY_NUM-1 has a specific certificate type associated with it:
Lines 381 to 390 in 3c957bc
So at position 0 in the pkeys array we will store information about an RSA cert. In position 2 we will store DSA certs, in position 7 Ed25519 certs. There are 9 different types in total that we could store. However in order to have a pluggable signature scheme we're going to have to be able to support signature algorithms that are unknown to libssl at compile time. They're added dynamically at run time, after we know what provider signature schemes we've plugged in.
Thanks for the detailed answers. We will rethink our approach with version 3.0.0, having the discussion above in mind.
In the OpenQuantumSafe project we have a research-level, proof-of-concept OSSL1.1.1 (fork) integration of all current NIST PQC candidates (KEMs+SIGs, incl. hybrids) but now look at moving this to OSSL3 to make our work more interesting for future uptake.
Before duplicating any effort, can I ask whether any progress has been made on the two topics discussed in this issue: a) a OSSL3 PQC integration (asked about by @MAKED0N ) and b) a fully pluggable new crypto provider facility in OSSL3 (suggested by @mattcaswell )? If I read the latest OSSL3 code correctly, Providers (certainly beyond KEMs) still are only meant to integrate alternative code for existing libcrypto implementations, but not an entirely new crypto "family" like PQC (again, SIG+KEM), right?
As this is the only OpenSSL issue mentioning "PQC", any current pointers to what the OSSL team is "up to" regarding PQC (beyond what is discussed here regarding KEMs ) would be welcome to guide our efforts.
In the OpenQuantumSafe project we have a research-level, proof-of-concept OSSL1.1.1 (fork) integration of all current NIST PQC candidates (KEMs+SIGs, incl. hybrids) but now look at moving this to OSSL3 to make our work more interesting for future uptake.
Before duplicating any effort, can I ask whether any progress has been made on the two topics discussed in this issue: a) a OSSL3 PQC integration (asked about by @MAKED0N ) and b) a fully pluggable new crypto provider facility in OSSL3 (suggested by @mattcaswell )? If I read the latest OSSL3 code correctly, Providers (certainly beyond KEMs) still are only meant to integrate alternative code for existing libcrypto implementations, but not an entirely new crypto "family" like PQC (again, SIG+KEM), right?
Yes, providers are just a way to provide alternate implementations (including new algorithms) for a given "family" of cryptographic primitive (like signature, digest, MAC, KEM, etc.), but requires libcrypto to support that "family" of operation.
As this is the only OpenSSL issue mentioning "PQC", any current pointers to what the OSSL team is "up to" regarding PQC (beyond what is discussed here regarding KEMs ) would be welcome to guide our efforts.
AFAIK the OpenSSL team is trying to focus on getting the 3.0 release ready and is not really doing anything directly with PQC (other than the KEM bits you already found).
If I read the latest OSSL3 code correctly, Providers (certainly beyond KEMs) still are only meant to integrate alternative code for existing libcrypto implementations, but not an entirely new crypto "family" like PQC (again, SIG+KEM), right?
I'm not sure if I'm reading your question correctly. Providers are fully intended to implement new algorithms that have not been implemented by libcrypto previously. But they do have to fit into existing operations, such as a signature scheme, or a digest scheme, or KEX scheme or a KEM scheme etc. If you had some new type of operation that we hadn't had before then we wouldn't be able to support that.
But implementing such a new algorithm would only make it available to libcrypto not libssl. We have implemented fully pluggable KEM and KEX in libssl, but haven't extended that to other operations such as signatures (which is the subject of this issue). There is a desire to do that but it is unlikely to happen in 3.0 timescale.
Thanks for the feedback & insight regarding your plans & prios. We'll then proceed with first priority with a PQC KEM-only provider using liboqs for OSSL3 and later a SIG-integration, possibly equally libssl pluggable (extending on provider-signature ), but probably initially only as a branch/patch to libssl.
I'm wonder if I could bounce some ideas here. I was thinking that struct ssl_ctx_st 's sigalg_lookup_cache could be made to be allowed to grow and then providers could be queried for a new capability called "TLS-SIGALG" . If the provider supported such a capability it would call the callback with OSSL_DISPATCH arrays giving enough information to fill in a struct sigalg_lookup_st .
This is of course insufficient as much work would still need to be done with s->cert->pkeys . But I'm asking this to solicit some opinions on whether this is a good approach?
I'm wonder if I could bounce some ideas here. I was thinking that struct ssl_ctx_st 's sigalg_lookup_cache could be made to be allowed to grow and then providers could be queried for a new capability called "TLS-SIGALG" . If the provider supported such a capability it would call the callback with OSSL_DISPATCH arrays giving enough information to fill in a struct sigalg_lookup_st .
This is of course insufficient as much work would still need to be done with s->cert->pkeys . But I'm asking this to solicit some opinions on whether this is a good approach?
This is broadly in line with my own thoughts on the matter and roughly equivalent to how we implemented the existing pluggable KEX/KEM functionality. I would very much expect something along similar lines.
There is the additional problem that any signature scheme would not only have to plug in to libssl, but would also have to plug in to the X509 code, i.e. not only would we have to do something with s->cert->pkeys to be able to handle keys for algorithms that we don't know about, but the X509 verification logic would also have to handle a signature OID that it doesn't know. That isn't quite possible yet although I know that @levitte has some thoughts on the matter and I've had numerous discussions with him on the topic. Another complication is handling AlgorithmIdentifiers that have parameters (currently the only one like this we have to deal with is RSA-PSS but in theory any algorithm could have this, e.g. see section 3.1 of RFC4055).
@mattcaswell, if I understand things correctly, there are two aspects of "pluggable" discussed here, and the one that only affects libssl is, how can a provider plug its own cipher suite into this (and similar) table:
Line 1023 in f839361
Also, most of all, we will need to reform that table when we get the functionality that you and I have discussed.
I totally see provider capabilities as a way to plug things into that table, and if I'm reading this right, that's exactly what the "TLS-SIGALG" idea is about.
As for X.509 processing, I believe that's covered as soon as signature algorithms can be fetched directly by OID, and we can pass the AlgID params, or the full AlgID to the fetched method.
@levitte :
I think we are on the same page here in terms of SSL.
I think the lookup cache will be quite simple, but the bigger work is modifying s->cert->pkeys[] . It also affects validity_flags and oddly enough cert_filename[] in struct ssl_conf_ctx_st .
the one that only affects libssl is, how can a provider plug its own cipher suite into this (and similar) table:
With the correction that you meant "signature algorithm" not "cipher suite" here - you are correct.
With the correction that you meant "signature algorithm" not "cipher suite" here - you are correct.
No, I did mean sipher suite, just pointed at the wrong table. I probably meant the SSL_CIPHER tables.
As for sigalg_lookup_tbl, I believe we should be able to throw it away as soon as we have directly fetchable sigalgs
No, I did mean sipher suite, just pointed at the wrong table. I probably meant the SSL_CIPHER tables.
Ciphersuites don't come into it. At least not in TLSv1.3 anyway which is where I think we should be concentrating. The ciphersuite says nothing about the sigalg at all in TLSv1.3.
As for sigalg_lookup_tbl, I believe we should be able to throw it away as soon as we have directly fetchable sigalgs
We will need some form of table (dynamically generated from the capabilities) of the sig algs that we support. For example a client needs to send a list of all the sig algs it supports to the server.
We will need some form of table (dynamically generated from the capabilities) of the sig algs that we support. For example a client needs to send a list of all the sig algs it supports to the server.
Ok. For provider supported sigalgs (and in a future where sigalgs are directly fetchable), it should be as simple as going through all the signature methods from a EVP_SIGNATURE_do_all_provided() call, and extract whatever we need from that.
For provider supported sigalgs (and in a future where sigalgs are directly fetchable), it should be as simple as going through all the signature methods from a EVP_SIGNATURE_do_all_provided() call, and extract whatever we need from that.
We will also need to know the IANA TLS signature scheme id:
https://www.iana.org/assignments/tls-parameters/tls-parameters.xhtml#tls-signaturescheme
We will also need to know the IANA TLS signature scheme id:
Yes? I was thinking that could be part of "... and extract whatever we need from that". The signature scheme id could be a gettable implementation parameter.
The signature scheme id could be a gettable implementation parameter.
Yes, that could work.
I've written some stuff that conforms to the discussion here and once I polish it up a bit I'm thinking of posting the diff here for your comments. I don't want to do it as a pull request because I know this is not a priority for 3.0.0 and most likely won't go into master branch anytime soon. Would a diff be appropriate?
Would a diff be appropriate?
Rather I would create a "draft" PR so that it is clear that it not ready for consideration for approval yet.
Please see #15599
After my short-sighted attempt in #18387 I now scoured the code more thoroughly and hopefully now understand things a bit better: To deliver pluggable TLS signatures it seems clear that a provider needs to deliver on a new provider capability, i.e., adding to include/openssl/core_names.h as already suggested above, e.g.:
/* SignatureScheme as per RFC 8446, 4.2.3 */
#define OSSL_CAPABILITY_TLS_SIGNATURE_SCHEME "tls-signature-scheme"
Then this information shall be queried along similar lines to pluggable groups, filling in sigalg_lookup_cache, ideally also filling a new "cert_lookup_cache" (instead of the overly static ssl_cert_info). This could be done in ssl_setup_sig_algs.
If OK (? @mattcaswell @levitte ?) what to do with the different instances of arrays of length "SSL_PKEY_NUM"? They could be made into dynamically extensible structures (by number of PKEY algorithms found at startup in all providers), but I wonder whether this is really necessary: Why are there these arrays to begin with? Shouldn't be there only a single cert active at any one time? Why then the pkeys array? Why the valid flags array? Wouldn't a single "current_CERT_PKEY" pointer suffice (possibly NULL if no valid cert available)? Or is the idea that all PKEY algorithm families shall be readily configurable and usable (from startup) at the same time such that a server can serve any (authentication) PKEY type requested by a client? If so, then I don't see an alternative to extensible structures instead of the fixed-length arrays currently indexed by "SSL_PKEY_NUM". Do you?
Other suggestions/extensions to the thoughts above very welcome; without any, I'll set about to do these changes with the goal to get pluggable TLS sigs working. Just a bit nervous about the extent of these changes....
Shouldn't be there only a single cert active at any one time?
Only after ciphersuite/sigalg negotiation. Prior to that it is perfectly possible to have multiple certs/keys. For example it is common to have both an ECDSA and an RSA cert active. If a (TLSv1.2) ECDSA based ciphersuite gets negotiated then the ECDSA cert is returned, otherwise of a (TLSv1.2) RSA based ciphersuite gets negotiated then the RSA cert is returned. It's also possible to have multiple certs in TLSv1.3
There is an important difference between TLSv1.3 and TLSv1.2 here. In 1.3 the cert selection is based on the sigalg. In 1.2 it is based on the ciphersuite. I'd suggest any "pluggable signatures" solution should focus on TLSv1.3. Plugging in a whole new ciphersuite would be quite a lot of additional work.
One possibility is to have both the static array and a dynamic array. The static array could be for well known types and would largely mean the TLSv1.2 code would just work as-is. The dynamic array could be filled in with new algorithms that we don't have built-in. Only TLSv1.3 cert selection would have to worry about those.
Why the valid flags array?
This is used during sig algs processing. We loop through the sig algs that are shared between both peers and mark which certificates are valid for use based on those sig algs. This guides cert/ciphersuite selection.
Lines 2354 to 2377 in 7e5e911
Thanks for the quick feedback!
I'd suggest any "pluggable signatures" solution should focus on TLSv1.3.
Absolutely. I wasn't eager to support TLS<1.3, really. Thought about also using something equivalent to OSSL_CAPABILITY_TLS_GROUP_MIN_TLS and enforcing it being >=TLS1_3.
One possibility is to have both the static array and a dynamic array.
That may be a solution; what array are you referring-to, though? pkeys, ssl_cert_info, cert_filename? Any other?
Could (or even should) sig_idx be (re-)used (larger values than "SSL_PKEY_NUM" pointing to the dynamic array)? Or should we also extend SIGALG_LOOKUP to leave "sig_idx"-based logic unchanged? I'm a bit nervous reading things like TLS 1.2 callers can override sig->sig_idx, but not TLS 1.3 callers. ... Also, tls1_process_sigalgs would need changing if we introduce something beyond "sig_idx": Do you mean that by
Only TLSv1.3 cert selection would have to worry about those.
?
That may be a solution; what array are you referring-to, though? pkeys, ssl_cert_info, cert_filename?
Most likely all of the above. Although ssl_cert_info looks like it would need further rework since the first field a fixed "nid" - which dynamically added algorithms don't have, and the second field is one of the SSL_a* fields. These are currently fixed values - but a pluggable sigalgs mechanism would have to be to add new ones.
Any other?
valid_flags
Or should we also extend SIGALG_LOOKUP to leave "sig_idx"-based logic unchanged? I'm a bit nervous reading things like TLS 1.2 callers can override sig->sig_idx, but not TLS 1.3 callers. .
There are quite a few places where we use the sig->sig_idx value to look a value up in one of the arrays. If we have both a static and dynamic array we will have to change all of those locations to look things up in the right location.
lso, tls1_process_sigalgs would need changing if we introduce something beyond "sig_idx":
Yes absolutely...and everywhere else we use sig_idx.
Only TLSv1.3 cert selection would have to worry about those.
TLSv1.3 cert selection is driven by the sigalgs. See find_sig_alg.
fixed "nid" - which dynamically added algorithms don't have
Hmm -- don't they? Those of oqsprovider do: We're (ultimately) calling core_obj_create and core_obj_add_sigid in order to get the new sig alg's OIDs registered, thus making the OBJ_... functions work for the new algorithms. This in turn means that OBJ_sn2nid delivers a NID for each new algorithm. Thus, "SIGALG_LOOKUP" should be usable as-is, no? Is that an unintended use of those APIs?
Thanks for the other pointers -- will heed them.
Hmm -- don't they? Those of oqsprovider do: We're (ultimately) calling core_obj_create and core_obj_add_sigid in order to get the new sig alg's OIDs registered, thus making the OBJ_... functions work for the new algorithms. This in turn means that OBJ_sn2nid delivers a NID for each new algorithm. Thus, "SIGALG_LOOKUP" should be usable as-is, no? Is that an unintended use of those APIs?
Ah, right! Yes good point. I forgot that providers can add these themselves.
One possibility is to have both the static array and a dynamic array.
That may be a solution; what array are you referring-to, though? pkeys, ssl_cert_info, cert_filename?
Most likely all of the above.
Just to quickly touch base as to whether I'm on the right track:
In order to retain CERT.pkeys (and necessarily CERT.key) as per the suggestion above I added a separate CERT_PKEY provider_cpkeys[] (initialized via the new provider capability "TLS-SIGNATURES") as well as a "selecting" provider_cpkeys_valid flag to the CERT structure.
Enabling this provider-driven CERT (array) to operate while retaining the semantics of the rest of CERT for processing based on (static/legacy)CERT.pkeys and CERT.key lots of changes throughout the sslfolder are required: Is that expected? If so, I'll surely carry on but the PR will be really large...
The question forming in the back of my mind: Might it not be less code change to make CERT.pkeys dynamic, with legacy PKEYs in the first positions and all provider-implemented PKEY types following? That way, the CERT.key selection logic everywhere could remain unchanged. SSL_PKEY_NUM of course would not be a constant any more.... Another benefit could be that eventually the default provider could (also) populate the new, extensible CERT.pkeys array, completely doing away with "static/legacy" handling for RSA, ECC, GOST, ...
The question forming in the back of my mind: Might it not be less code change to make CERT.pkeys dynamic, with legacy PKEYs in the first positions and all provider-implemented PKEY types following?
That could work - I'm open to that as a possibility.
Either way I suspect there will be a lot of changes.
We do have some code in the TLS extensions-processing logic that handles a "raw_extensions" array that comprises an initial fixed list of built-in extensions followed by any custom extensions supported. That works ... okay, and might be a rough model for what this would look like.
It might also be worth considering keeping a static array and using a non-array structure (STACK_OF?) for the dynamic things. But that would probably be quite invasive, based on the discussion above.
Either way I suspect there will be a lot of changes.
That is confirmed now after working quite a bit on this.
To be safe I'm not doing silly things, particularly concerning ABI compatibility can you please confirm that any changes to "ssl/ssl_local.h" (structures, function parameters, incl. "const"-ness) are OK -- as long as they have no impact on/are not changing anything in "include/openssl"?
As a "worked example", it seems necessary to pass a parameter with the max numbers of PKEYs (now known only after loading providers) to ssl_cert_new (to do away with simply using the global constant SSL_PKEY_NUM):
-> Is my understanding correct that adding such parameter to ssl_cert_new is OK? Or would I have to create a new function ssl_cert_new_ex to retain API and ABI of the original function (even though it would not be used in the code any more -- but is a non-static function that someone may link to)?
Anything outside include/openssl is fine to change. But of course also the behavior of the functions in include/openssl cannot change in backwards incompatible way.
And in regards the provider interfaces they must be compatible in backwards and forwards ways. I.e., new providers must be able to work with old libcrypto/libssl and the other way around as well.
Anything outside include/openssl is fine to change. But of course also the behavior of the functions in include/openssl cannot change in backwards incompatible way.
Thanks for the confirmation. Your latter statement also re-confirms my decision to leave "intact" everything labelled "legacy", even though there are quite a few comments already in the code wondering about removal of such code. In any case, all this aligns with my goal of creating as small as possible a PR.
There's now a working implementation of all the changes discussed above. I'd be grateful if someone would find the time to glance over the logic/conceptual changes: https://github.com/openssl/openssl/compare/master...baentsch:openssl:sigload?expand=1 . If this should be done via draft PR, please let me know.
Also welcome would be general logic improvement suggestions (and a question whether the use of size_t as a type should be avoided given the many type-checking CI "errors" triggered in int loops -- or whether the loops should be moved to use size_t iteration vars).
Further background: As per the discussion above, changes are kept to the minimum: legacy alg logic was maintained to the max and none of the built-in providers got the corresponding "enablement changes". oqs-provider was suitably enhanced though and in conjunction with this update tests successfully all QSC algorithms with "traditional" signature sizes:
None of the OpenSSL size constants returned by ossl_statem_client_max_message_size and ossl_statem_server_max_message_size were changed to make the algorithms above work, but a discussion on this topic may be needed as the failures above all are triggered by size checks in the two functions above: My suggestion would be to defer this until this issue is closed, though.
And in regards the provider interfaces they must be compatible in backwards and forwards ways. I.e., new providers must be able to work with old libcrypto/libssl and the other way around as well.
I don't agree with this. Our providers require this. 3rd party ones don't have do. I see no problem with someone else's provider mandating 3.0.5 or later. If we later incorporate such a provider into OpenSSL, I still see no reason to mandate backwards compatibility.
And in regards the provider interfaces they must be compatible in backwards and forwards ways. I.e., new providers must be able to work with old libcrypto/libssl and the other way around as well.
I don't agree with this. Our providers require this. 3rd party ones don't have do. I see no problem with someone else's provider mandating 3.0.5 or later. If we later incorporate such a provider into OpenSSL, I still see no reason to mandate backwards compatibility.
Agreed, external providers are allowed to do what they want in this regard.
Perhaps we should be much more careful when we talk about this sort of stuff, and explicitly repeat when we're talking about our own provs. It's obvious that we're expressing things a bit too vaguely at times...
There's now a working implementation of all the changes discussed above. I'd be grateful if someone would find the time to glance over the logic/conceptual changes: https://github.com/openssl/openssl/compare/master...baentsch:openssl:sigload?expand=1 . If this should be done via draft PR, please let me know.
This is cool!
I've glanced over it. It needs a more detailed and thorough review, but conceptually this is looking good. When you are ready a draft PR would be a good idea so that I can more easily provide review comments.
None of the OpenSSL size constants returned by ossl_statem_client_max_message_size and ossl_statem_server_max_message_size were changed to make the algorithms above work, but a discussion on this topic may be needed as the failures above all are triggered by size checks in the two functions above: My suggestion would be to defer this until this issue is closed, though.
Those constants are just to sanity check the sizes. We've never needed or supported larger sizes before, so it made sense. If the constants no longer represent what is sane then we can change them. I agree though that we can defer that until this issue is closed.
I've glanced over it. It needs a more detailed and thorough review, but conceptually this is looking good. When you are ready a draft PR would be a good idea so that I can more easily provide review comments.
Thanks for the quick initial feedback! Will do a draft PR -- but before would like to get rid of the many CI errors of the same type:
ssl/ssl_cert.c:371:21: error: comparison of integer expressions of different signedness: 'int' and 'size_t' {aka 'long unsigned int'} [-Werror=sign-compare]
371 | for (i = idx; i < c->ssl_pkey_num; i++) {
| ^
--> What's your suggestion/preference here: a) Change type of new variable ssl_pkey_num (et.al.) to int to silence the compiler complaints or b) change (factually positive) loop variable declarations all over to size_t or c) add casts like here?
My preference is (b).
My preference is (b).
So implemented. Comments now solicited via #19312 .
Suggest to close this issue as #19312 landed.