In OpenSSL 3, ASN1_item_verify*() can return 2 on error.

[Theo Buehler]

Introduced in #10942 as part of extracting ASN1_item_verify_ctx() from ASN1_item_verify() when a ret = -1; was deleted after the goto err here:

openssl/crypto/asn1/a_verify.c

Lines 143 to 154 in 5bbdbce

         ret = pkey->ameth->item_verify(ctx, it, data, alg, signature, pkey); 
         /* 
          * Return values meaning: 
          * <=0: error. 
          *   1: method does everything. 
          *   2: carry on as normal, method has called EVP_DigestVerifyInit() 
          */ 
         if (ret <= 0) 
             ERR_raise(ERR_LIB_ASN1, ERR_R_EVP_LIB); 
         if (ret <= 1) 
             goto err; 
     } else { 

If item_verify() returns 2 (which it usually will for RSA-PSS for example) the control flow would skip here and return 2 if ASN1_item_i2d() errors:

openssl/crypto/asn1/a_verify.c

Lines 202 to 210 in 5bbdbce

 inl = ASN1_item_i2d(data, &buf_in, it); 
 if (inl <= 0) { 
     ERR_raise(ERR_LIB_ASN1, ERR_R_INTERNAL_ERROR); 
     goto err; 
 } 
 if (buf_in == NULL) { 
     ERR_raise(ERR_LIB_ASN1, ERR_R_ASN1_LIB); 
     goto err; 
 } 

[Tomáš Mráz]

Fix in #24576