test_pkey_ec.rb test failures in OpenSSL FIPS
junaruga opened this issue · comments
I am trying to fix the test failures in test/openssl/test_pkey_ec.rb now in OpenSSL FIPS on the ruby/openssl latest master branch f4b8dacc75d61142b7b4e0142898b2fecbb131b9, and openssl/openssl latest master branch cf712830b7b5a20a768a1fc5f78dc48841b7617f.
Test failures
$ OPENSSL_CONF=/home/jaruga/.local/openssl-3.2.0-dev-fips-debug-cf712830b7/ssl/openssl_fips.cnf \
bundle exec rake debug
...
ruby 3.3.0dev (2023-05-30T12:39:26Z master 30b960ba34) [x86_64-linux]
OpenSSL::OPENSSL_VERSION: OpenSSL 3.2.0-dev
OpenSSL::OPENSSL_LIBRARY_VERSION: OpenSSL 3.2.0-dev
OpenSSL::OPENSSL_VERSION_NUMBER: 30200000
OpenSSL::LIBRESSL_VERSION_NUMBER: undefined
FIPS enabled: true
$ OPENSSL_CONF=/home/jaruga/.local/openssl-3.2.0-dev-fips-debug-cf712830b7/ssl/openssl_fips.cnf \
ruby -I./lib -ropenssl test/openssl/test_pkey_ec.rb
Loaded suite test/openssl/test_pkey_ec
Started
E
===================================================================================================================================================================================================================
Error: test_ECPrivateKey_encrypted(OpenSSL::TestEC): OpenSSL::PKey::ECError: invalid curve name
test/openssl/test_pkey_ec.rb:247:in `initialize'
test/openssl/test_pkey_ec.rb:247:in `new'
test/openssl/test_pkey_ec.rb:247:in `test_ECPrivateKey_encrypted'
244: 0/dGSU5SzFG+iT9iFXCwCvv+bxyegkBOyALFje1NAsM=
245: -----END EC PRIVATE KEY-----
246: EOF
=> 247: key = OpenSSL::PKey::EC.new(pem, "abcdef")
248: assert_same_ec p256, key
249: key = OpenSSL::PKey::EC.new(pem) { "abcdef" }
250: assert_same_ec p256, key
===================================================================================================================================================================================================================
E
===================================================================================================================================================================================================================
Error: test_ec_key(OpenSSL::TestEC): NoMethodError: undefined method `filter_backtrace' for module Test
/home/jaruga/var/git/ruby/openssl/test/lib/core_assertions.rb:188:in `block in assert_nothing_raised'
/home/jaruga/var/git/ruby/openssl/test/lib/core_assertions.rb:26:in `block in message'
/home/jaruga/var/git/ruby/openssl/test/lib/core_assertions.rb:190:in `rescue in assert_nothing_raised'
/home/jaruga/var/git/ruby/openssl/test/lib/core_assertions.rb:181:in `assert_nothing_raised'
test/openssl/test_pkey_ec.rb:19:in `block in test_ec_key'
16: key = OpenSSL::PKey::EC.generate(curve_name)
17: assert_predicate key, :private?
18: assert_predicate key, :public?
=> 19: assert_nothing_raised { key.check_key }
20: end
21:
22: key1 = OpenSSL::PKey::EC.generate("prime256v1")
test/openssl/test_pkey_ec.rb:11:in `each'
test/openssl/test_pkey_ec.rb:11:in `test_ec_key'
===================================================================================================================================================================================================================
Finished in 0.044746681 seconds.
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
16 tests, 124 assertions, 0 failures, 2 errors, 0 pendings, 0 omissions, 0 notifications
87.5% passed
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
357.57 tests/s, 2771.16 assertions/s
A minimal reproducer
For the test/openssl/test_pkey_ec.rb:19, below is a minimal reproducer.
$ OPENSSL_CONF=/home/jaruga/.local/openssl-3.2.0-dev-fips-debug-cf712830b7/ssl/openssl_fips.cnf \
ruby -I./lib -ropenssl -e 'OpenSSL::PKey::EC.generate("secp112r1").check_key'
-e:1:in `check_key': EVP_PKEY_check: initialization error (OpenSSL::PKey::ECError)
from -e:1:in `<main>'
Debug with GDB
$ OPENSSL_CONF=/home/jaruga/.local/openssl-3.2.0-dev-fips-debug-cf712830b7/ssl/openssl_fips.cnf \
gdb --args ruby -I./lib -ropenssl -e 'OpenSSL::PKey::EC.generate("secp112r1").check_key'
...
(gdb) set environment LD_LIBRARY_PATH /home/jaruga/.local/openssl-3.2.0-dev-fips-debug-cf712830b7/lib
(gdb) b ossl_ec_key_check_key
(gdb) r
...
(gdb) n
551 if (EVP_PKEY_check(pctx) != 1) {
(gdb) p EVP_PKEY_check(pctx)
$1 = 0
(gdb) f
#0 ossl_ec_key_check_key (self=140737044094120)
at ../../../../ext/openssl/ossl_pkey_ec.c:551
551 if (EVP_PKEY_check(pctx) != 1) {
(gdb) n
552 EVP_PKEY_CTX_free(pctx)
(gdb) p pctx
$2 = (EVP_PKEY_CTX *) 0x7e17d0
(gdb) p *pctx
$3 = {operation = 0, libctx = 0x0, propquery = 0x0,
keytype = 0x7fffe54c830a "id-ecPublicKey", keymgmt = 0x7c5410, op = {keymgmt = {
genctx = 0x0}, kex = {exchange = 0x0, algctx = 0x0}, sig = {signature = 0x0,
algctx = 0x0}, ciph = {cipher = 0x0, algctx = 0x0}, encap = {kem = 0x0,
algctx = 0x0}}, cached_parameters = {dist_id_name = 0x0, dist_id = 0x0,
dist_id_len = 0, dist_id_set = 0}, app_data = 0x0, pkey_gencb = 0x0,
keygen_info = 0x0, keygen_info_count = 0, legacy_keytype = 408, pmeth = 0x0,
engine = 0x0, pkey = 0x7e0570, peerkey = 0x0, data = 0x0, flag_call_digest_custom = 0,
rsa_pubexp = 0x0}
(gdb) n
553 ossl_raise(eECError, "EVP_PKEY_check");
The EVP_PKEY_check(pctx) returns 0 in the line below. And it seems that causes the EVP_PKEY_check: initialization error (OpenSSL::PKey::ECError). Do you know why this happens?
openssl/ext/openssl/ossl_pkey_ec.c
Lines 551 to 554 in f4b8dac
Error: test_ec_key(OpenSSL::TestEC): NoMethodError: undefined method `filter_backtrace' for module Test
/home/jaruga/var/git/ruby/openssl/test/lib/core_assertions.rb:188:in `block in assert_nothing_raised'
For the error above, where does the Test.filter_backtrace come from?
openssl/test/lib/core_assertions.rb
Lines 185 to 190 in f4b8dac
For the error above, where does the
Test.filter_backtracecome from?
The test/lib/core_assertions.rb including the Test.filter_backtrace was added by the 520601e by @hsbt. @hsbt, do you know why the error by the Test.filter_backtrace happens?
And I am seeing the same name method filter_backtrace in the test-unit. Is it just co-incidence?
https://github.com/test-unit/test-unit/blob/8cd57617a3cb470985cf5768691e10ca72928a1c/lib/test/unit/util/backtracefilter.rb#L20-L55
$ OPENSSL_CONF=/home/jaruga/.local/openssl-3.2.0-dev-fips-debug-cf712830b7/ssl/openssl_fips.cnf \ ruby -I./lib -ropenssl -e 'OpenSSL::PKey::EC.generate("secp112r1").check_key' -e:1:in `check_key': EVP_PKEY_check: initialization error (OpenSSL::PKey::ECError) from -e:1:in `<main>'
secp112r1 isn't allowed in FIPS 140.
Error: test_ec_key(OpenSSL::TestEC): NoMethodError: undefined method `filter_backtrace' for module Test /home/jaruga/var/git/ruby/openssl/test/lib/core_assertions.rb:188:in `block in assert_nothing_raised'For the error above, where does the
Test.filter_backtracecome from?openssl/test/lib/core_assertions.rb
Lines 185 to 190 in f4b8dac
First, we should switch to using https://github.com/ruby/test-unit-ruby-core instead of embedding core_assertions.rb. This is just not yet worked on.
The same issue is in ruby/test-unit-ruby-core. Test.filter_backtrace appears to be defined in ruby/ruby's tool/lib/test/unit.rb. It could be included in, too, but there doesn't seem a need for CoreAssertions to define assert_nothing_raised since test-unit already provides this assertion method.
ruby/test-unit-ruby-core#2 (comment) is helpful for openssl?
Yes, test-unit-ruby-core 1.0.2 fixed assert_nothing_raised for me. PR #673 lets this repository use the gem instead of embedding core_assertions.rb.
@hsbt @rhenium thanks! On the latest ruby/opessl including the #673, the assert_nothing_raised fails properly.
$ OPENSSL_CONF=/home/jaruga/.local/openssl-3.2.0-dev-fips-debug-cf712830b7/ssl/openssl_fips.cnf \
bundle exec ruby -I./lib -ropenssl test/openssl/test_pkey_ec.rb
Loaded suite test/openssl/test_pkey_ec
Started
E
==========================================================================================
Error: test_ECPrivateKey_encrypted(OpenSSL::TestEC): OpenSSL::PKey::ECError: invalid curve name
test/openssl/test_pkey_ec.rb:247:in `initialize'
test/openssl/test_pkey_ec.rb:247:in `new'
test/openssl/test_pkey_ec.rb:247:in `test_ECPrivateKey_encrypted'
244: 0/dGSU5SzFG+iT9iFXCwCvv+bxyegkBOyALFje1NAsM=
245: -----END EC PRIVATE KEY-----
246: EOF
=> 247: key = OpenSSL::PKey::EC.new(pem, "abcdef")
248: assert_same_ec p256, key
249: key = OpenSSL::PKey::EC.new(pem) { "abcdef" }
250: assert_same_ec p256, key
==========================================================================================
F
==========================================================================================
Failure: test_ec_key(OpenSSL::TestEC):
Exception raised:
<#<OpenSSL::PKey::ECError: EVP_PKEY_check: initialization error>>
Backtrace:
test/openssl/test_pkey_ec.rb:19:in `check_key'
test/openssl/test_pkey_ec.rb:19:in `block (2 levels) in test_ec_key'.
test/openssl/test_pkey_ec.rb:19:in `check_key'
test/openssl/test_pkey_ec.rb:19:in `block (2 levels) in test_ec_key'
16: key = OpenSSL::PKey::EC.generate(curve_name)
17: assert_predicate key, :private?
18: assert_predicate key, :public?
=> 19: assert_nothing_raised { key.check_key }
20: end
21:
22: key1 = OpenSSL::PKey::EC.generate("prime256v1")
/home/jaruga/var/git/ruby/openssl/bundle/ruby/3.3.0+0/gems/test-unit-ruby-core-1.0.2/lib/core_assertions.rb:219:in `assert_nothing_raised'
test/openssl/test_pkey_ec.rb:19:in `block in test_ec_key'
test/openssl/test_pkey_ec.rb:11:in `each'
test/openssl/test_pkey_ec.rb:11:in `test_ec_key'
==========================================================================================
Finished in 0.050731255 seconds.
------------------------------------------------------------------------------------------
16 tests, 124 assertions, 1 failures, 1 errors, 0 pendings, 0 omissions, 0 notifications
87.5% passed
------------------------------------------------------------------------------------------
315.39 tests/s, 2444.25 assertions/s
$ OPENSSL_CONF=/home/jaruga/.local/openssl-3.2.0-dev-fips-debug-cf712830b7/ssl/openssl_fips.cnf \ ruby -I./lib -ropenssl -e 'OpenSSL::PKey::EC.generate("secp112r1").check_key' -e:1:in `check_key': EVP_PKEY_check: initialization error (OpenSSL::PKey::ECError) from -e:1:in `<main>'secp112r1 isn't allowed in FIPS 140.
How did you know that? Could you share a document link that you checked for that?
secp112r1 isn't allowed in FIPS 140.
How did you know that? Could you share a document link that you checked for that?
Is it https://www.openssl.org/source/ - The OpenSSL 3.0.0 or 3.0.8 security policy document?
secp112r1, defined on a 112-bit finite field, would provide at most 56 bits of security. I don't know how to cite the specifications for FIPS 140, but it must definitely be prohibited for any use.
The test case in question is about OpenSSL::PKey::EC.builtin_curves. I don't think the current assertions make much sense anyway. I think it can use something like:
diff --git a/test/openssl/test_pkey_ec.rb b/test/openssl/test_pkey_ec.rb
index e5fef940a6c3..ab777a8b48a4 100644
--- a/test/openssl/test_pkey_ec.rb
+++ b/test/openssl/test_pkey_ec.rb
@@ -5,20 +5,6 @@
class OpenSSL::TestEC < OpenSSL::PKeyTestCase
def test_ec_key
- builtin_curves = OpenSSL::PKey::EC.builtin_curves
- assert_not_empty builtin_curves
-
- builtin_curves.each do |curve_name, comment|
- # Oakley curves and X25519 are not suitable for signing and causes
- # FIPS-selftest failure on some environment, so skip for now.
- next if ["Oakley", "X25519"].any? { |n| curve_name.start_with?(n) }
-
- key = OpenSSL::PKey::EC.generate(curve_name)
- assert_predicate key, :private?
- assert_predicate key, :public?
- assert_nothing_raised { key.check_key }
- end
-
key1 = OpenSSL::PKey::EC.generate("prime256v1")
# PKey is immutable in OpenSSL >= 3.0; constructing an empty EC object is
@@ -49,6 +35,17 @@ def test_ec_key
end
end
+ def test_builtin_curves
+ builtin_curves = OpenSSL::PKey::EC.builtin_curves
+ assert_not_empty builtin_curves
+ assert_equal 2, builtin_curves[0].size
+ assert_kind_of String, builtin_curves[0][0]
+ assert_kind_of String, builtin_curves[0][1]
+
+ builtin_curve_names = builtin_curves.map { |name, comment| name }
+ assert_include builtin_curve_names, "prime256v1"
+ end
+
def test_generate
assert_raise(OpenSSL::PKey::ECError) { OpenSSL::PKey::EC.generate("non-existent") }
g = OpenSSL::PKey::EC::Group.new("prime256v1")secp112r1, defined on a 112-bit finite field, would provide at most 56 bits of security. I don't know how to cite the specifications for FIPS 140, but it must definitely be prohibited for any use.
The test case in question is about OpenSSL::PKey::EC.builtin_curves. I don't think the current assertions make much sense anyway. I think it can use something like:
diff --git a/test/openssl/test_pkey_ec.rb b/test/openssl/test_pkey_ec.rb index e5fef940a6c3..ab777a8b48a4 100644 --- a/test/openssl/test_pkey_ec.rb +++ b/test/openssl/test_pkey_ec.rb @@ -5,20 +5,6 @@ class OpenSSL::TestEC < OpenSSL::PKeyTestCase def test_ec_key - builtin_curves = OpenSSL::PKey::EC.builtin_curves - assert_not_empty builtin_curves - - builtin_curves.each do |curve_name, comment| - # Oakley curves and X25519 are not suitable for signing and causes - # FIPS-selftest failure on some environment, so skip for now. - next if ["Oakley", "X25519"].any? { |n| curve_name.start_with?(n) } - - key = OpenSSL::PKey::EC.generate(curve_name) - assert_predicate key, :private? - assert_predicate key, :public? - assert_nothing_raised { key.check_key } - end - key1 = OpenSSL::PKey::EC.generate("prime256v1") # PKey is immutable in OpenSSL >= 3.0; constructing an empty EC object is @@ -49,6 +35,17 @@ def test_ec_key end end + def test_builtin_curves + builtin_curves = OpenSSL::PKey::EC.builtin_curves + assert_not_empty builtin_curves + assert_equal 2, builtin_curves[0].size + assert_kind_of String, builtin_curves[0][0] + assert_kind_of String, builtin_curves[0][1] + + builtin_curve_names = builtin_curves.map { |name, comment| name } + assert_include builtin_curve_names, "prime256v1" + end + def test_generate assert_raise(OpenSSL::PKey::ECError) { OpenSSL::PKey::EC.generate("non-existent") } g = OpenSSL::PKey::EC::Group.new("prime256v1")
I see. The solution looks okay to me. I may notice that one issue related to the OpenSSL::PKey::EC.builtin_curves (the used EC_get_builtin_curves). The issue is it seems to me that the result of the EC_get_builtin_curves(curves, crv_len) in the code below looks same in both FIPS and non-FIPS cases.
FIPS
$ OPENSSL_CONF=/home/jaruga/.local/openssl-3.2.0-dev-fips-debug-cf712830b7/ssl/openssl_fips.cnf \
ruby -I./lib -ropenssl -e 'p OpenSSL::PKey::EC.builtin_curves.size'
82
$ OPENSSL_CONF=/home/jaruga/.local/openssl-3.2.0-dev-fips-debug-cf712830b7/ssl/openssl_fips.cnf \
ruby -I./lib -ropenssl -e 'p OpenSSL::PKey::EC.builtin_curves[0]'
["secp112r1", "SECG/WTLS curve over a 112 bit prime field"]
Non-FIPS
$ ruby -I./lib -ropenssl -e 'p OpenSSL::PKey::EC.builtin_curves.size'
82
$ ruby -I./lib -ropenssl -e 'p OpenSSL::PKey::EC.builtin_curves[0]'
["secp112r1", "SECG/WTLS curve over a 112 bit prime field"]
This is weird to me. Because the EC_get_builtin_curves returns the curve_list.
And it seems that the curve_list is declared conditionally in both FIPS (FIPS_MODULE is defined) at crypto/ec/ec_curve.c#L2827 and non-FIPS (FIPS_MODULE not defined) crypto/ec/ec_curve.c#L2901. So, I expect the result of the EC_get_builtin_curves is different between FIPS and non-FIPS cases. I might see that FIPS_MODULE macro is not defined in the FIPS case. I plan to ask this to OpenSSL project.
This is weird to me. Because the
EC_get_builtin_curvesreturns thecurve_list.And it seems that the
curve_listis declared conditionally in both FIPS (FIPS_MODULEis defined) at crypto/ec/ec_curve.c#L2827 and non-FIPS (FIPS_MODULEnot defined) crypto/ec/ec_curve.c#L2901. So, I expect the result of theEC_get_builtin_curvesis different between FIPS and non-FIPS cases. I might see thatFIPS_MODULEmacro is not defined in the FIPS case. I plan to ask this to OpenSSL project.
The EC_get_builtin_curves() called from ruby/openssl would be the one in libcrypto.so. Since this function isn't provider-aware (rather, crypto/ec is the backend used by the default/fips providers), and considering it's possible to use FIPS module and non-FIPS module at the same time with custom property query string (which is however not currently supported in ruby/openssl - https://www.openssl.org/docs/man3.0/man7/fips_module.html), I don't think this is fixable.
The commit that introduced the defined(FIPS_MODULE) version of the list (openssl/openssl@10c2564, 2019) says it's for tests.
Thanks for the investigation. I found the issue ticket, openssl/openssl#18273 (comment) related to this topic, and commented there. It seems that the curve_list is not flexible.
Error: test_ECPrivateKey_encrypted(OpenSSL::TestEC): OpenSSL::PKey::ECError: invalid curve name
A minimal reproducer
For the test failure above, below is a minimal reproducer. Does the "AES-128-CBC" mean MD5? And it is not allowed in FIPS 140? I found the AES's Wikipedia page including the FIPS 140 things.
$ OPENSSL_CONF=/home/jaruga/.local/openssl-3.2.0-dev-fips-debug-cf712830b7/ssl/openssl_fips.cnf \
ruby -I./lib -ropenssl -e 'OpenSSL::PKey::EC.new("-----BEGIN EC PRIVATE KEY-----\nProc-Type: 4,ENCRYPTED\nDEK-Info: AES-128-CBC,85743EB6FAC9EA76BF99D9328AFD1A66\n\nnhsP1NHxb53aeZdzUe9umKKyr+OIwQq67eP0ONM6E1vFTIcjkDcFLR6PhPFufF4m\ny7E2HF+9uT1KPQhlE+D63i1m1Mvez6PWfNM34iOQp2vEhaoHHKlR3c43lLyzaZDI\n0/dGSU5SzFG+iT9iFXCwCvv+bxyegkBOyALFje1NAsM=\n-----END EC PRIVATE KEY-----\n", "abcdef")'
-e:1:in `initialize': invalid curve name (OpenSSL::PKey::ECError)
from -e:1:in `new'
from -e:1:in `<main>'
I referred to the following code.
openssl/test/openssl/test_pkey_ec.rb
Lines 237 to 247 in 7c34a43
Debug with GDB
FIPS case
$ OPENSSL_CONF=/home/jaruga/.local/openssl-3.2.0-dev-fips-debug-cf712830b7/ssl/openssl_fips.cnf \
gdb --args ruby -I./lib -ropenssl -e 'OpenSSL::PKey::EC.new("-----BEGIN EC PRIVATE KEY-----\nProc-Type: 4,ENCRYPTED\nDEK-Info: AES-128-CBC,85743EB6FAC9EA76BF99D9328AFD1A66\n\nnhsP1NHxb53aeZdzUe9umKKyr+OIwQq67eP0ONM6E1vFTIcjkDcFLR6PhPFufF4m\ny7E2HF+9uT1KPQhlE+D63i1m1Mvez6PWfNM34iOQp2vEhaoHHKlR3c43lLyzaZDI\n0/dGSU5SzFG+iT9iFXCwCvv+bxyegkBOyALFje1NAsM=\n-----END EC PRIVATE KEY-----\n", "abcdef")'
...
(gdb) set environment LD_LIBRARY_PATH /home/jaruga/.local/openssl-3.2.0-dev-fips-debug-cf712830b7/lib
(gdb) b ossl_ec_key_initialize
(gdb) r
...
(gdb) f
#0 ec_key_new_from_group (arg=140737044503040)
at ../../../../ext/openssl/ossl_pkey_ec.c:85
85 ossl_raise(eECError, "invalid curve name");
(gdb) bt
#0 ec_key_new_from_group (arg=140737044503040)
at ../../../../ext/openssl/ossl_pkey_ec.c:85
#1 0x00007fffe5802544 in ossl_ec_key_initialize (argc=2, argv=0x7fffe9eff048,
self=140737044094080) at ../../../../ext/openssl/ossl_pkey_ec.c:170
#2 0x00007ffff7cd34ff in ractor_safe_call_cfunc_m1 (recv=140737044094080, argc=2,
argv=0x7fffe9eff048, func=0x7fffe58023e4 <ossl_ec_key_initialize>)
at vm_insnhelper.c:3242
#3 0x00007ffff7ced42a in vm_call0_cfunc_with_frame (ec=0x40a180,
calling=0x7fffffffc120, argv=0x7fffe9eff048) at vm_eval.c:170
#4 0x00007ffff7ced5a9 in vm_call0_cfunc (ec=0x40a180, calling=0x7fffffffc120,
argv=0x7fffe9eff048) at vm_eval.c:184
#5 0x00007ffff7ced815 in vm_call0_body (ec=0x40a180, calling=0x7fffffffc120,
argv=0x7fffe9eff048) at vm_eval.c:230
#6 0x00007ffff7cecfd9 in vm_call0_cc (ec=0x40a180, recv=140737044094080, id=3137,
argc=2, argv=0x7fffe9eff048, cc=0x7fffe5851c58, kw_splat=0) at vm_eval.c:107
#7 0x00007ffff7cee89c in rb_call0 (ec=0x40a180, recv=140737044094080, mid=3137, argc=2,
argv=0x7fffe9eff048, call_scope=CALL_FCALL, self=140737042249120) at vm_eval.c:566
#8 0x00007ffff7cef573 in rb_call (recv=140737044094080, mid=3137, argc=2,
argv=0x7fffe9eff048, scope=CALL_FCALL) at vm_eval.c:892
#9 0x00007ffff7cefbfb in rb_funcallv_kw (recv=140737044094080, mid=3137, argc=2,
argv=0x7fffe9eff048, kw_splat=0) at vm_eval.c:1085
#10 0x00007ffff7af3f7f in rb_obj_call_init_kw (obj=140737044094080, argc=2,
argv=0x7fffe9eff048, kw_splat=0) at eval.c:1687
#11 0x00007ffff7b9b136 in rb_class_new_instance_pass_kw (argc=2, argv=0x7fffe9eff048,
klass=140737042249120) at object.c:2107
#12 0x00007ffff7cd34ff in ractor_safe_call_cfunc_m1 (recv=140737042249120, argc=2,
argv=0x7fffe9eff048, func=0x7ffff7b9b0f9 <rb_class_new_instance_pass_kw>)
at vm_insnhelper.c:3242
#13 0x00007ffff7cd414d in vm_call_cfunc_with_frame_ (ec=0x40a180,
reg_cfp=0x7fffe9ffef90, calling=0x7fffffffc730, argc=2, argv=0x7fffe9eff048,
stack_bottom=0x7fffe9eff040) at vm_insnhelper.c:3433
#14 0x00007ffff7cd4371 in vm_call_cfunc_with_frame (ec=0x40a180, reg_cfp=0x7fffe9ffef90,
calling=0x7fffffffc730) at vm_insnhelper.c:3461
#15 0x00007ffff7cd4482 in vm_call_cfunc_other (ec=0x40a180, reg_cfp=0x7fffe9ffef90,
calling=0x7fffffffc730) at vm_insnhelper.c:3487
#16 0x00007ffff7cd489a in vm_call_cfunc (ec=0x40a180, reg_cfp=0x7fffe9ffef90,
calling=0x7fffffffc730) at vm_insnhelper.c:3569
#17 0x00007ffff7cd6e2c in vm_call_method_each_type (ec=0x40a180, cfp=0x7fffe9ffef90,
calling=0x7fffffffc730) at vm_insnhelper.c:4327
#18 0x00007ffff7cd78d2 in vm_call_method (ec=0x40a180, cfp=0x7fffe9ffef90,
calling=0x7fffffffc730) at vm_insnhelper.c:4453
#19 0x00007ffff7cd7ad0 in vm_call_general (ec=0x40a180, reg_cfp=0x7fffe9ffef90,
calling=0x7fffffffc730) at vm_insnhelper.c:4497
#20 0x00007ffff7cda260 in vm_sendish (ec=0x40a180, reg_cfp=0x7fffe9ffef90, cd=0x7dd400,
block_handler=0, method_explorer=mexp_search_method) at vm_insnhelper.c:5487
#21 0x00007ffff7ce1c5b in vm_exec_core (ec=0x40a180, initial=0) at insns.def:835
#22 0x00007ffff7cf866c in rb_vm_exec (ec=0x40a180) at vm.c:2384
#23 0x00007ffff7cf94c6 in rb_iseq_eval_main (iseq=0x7fffe5851f28) at vm.c:2643
#24 0x00007ffff7af0c0b in rb_ec_exec_node (ec=0x40a180, n=0x7fffe5851f28) at eval.c:287
#25 0x00007ffff7af0d6c in ruby_run_node (n=0x7fffe5851f28) at eval.c:328
#26 0x00000000004011d9 in rb_main (argc=5, argv=0x7fffffffd898) at ./main.c:39
#27 0x0000000000401236 in main (argc=5, argv=0x7fffffffd898) at ./main.c:58
The error happens in the ec_key_new_from_group after failing to get the pkey from ossl_pkey_read_generic.
openssl/ext/openssl/ossl_pkey_ec.c
Lines 166 to 172 in 7c34a43
The error happens in the ec_key_new_from_group.
openssl/ext/openssl/ossl_pkey_ec.c
Line 85 in 7c34a43
Non-FIPS case
In the non-FIPS case, the ossl_pkey_read_generic can get the pkey unlike FIPS case.
$ gdb --args ruby -I./lib -ropenssl -e 'OpenSSL::PKey::EC.new("-----BEGIN EC PRIVATE KEY-----\nProc-Type: 4,ENCRYPTED\nDEK-Info: AES-128-CBC,85743EB6FAC9EA76BF99D9328AFD1A66\n\nnhsP1NHxb53aeZdzUe9umKKyr+OIwQq67eP0ONM6E1vFTIcjkDcFLR6PhPFufF4m\ny7E2HF+9uT1KPQhlE+D63i1m1Mvez6PWfNM34iOQp2vEhaoHHKlR3c43lLyzaZDI\n0/dGSU5SzFG+iT9iFXCwCvv+bxyegkBOyALFje1NAsM=\n-----END EC PRIVATE KEY-----\n", "abcdef")'
(gdb) set environment LD_LIBRARY_PATH /home/jaruga/.local/openssl-3.2.0-dev-fips-debug-cf712830b7/lib
(gdb) b ossl_ec_key_initialize
...
(gdb) n
166 pkey = ossl_pkey_read_generic(in, pass);
(gdb) n
167 BIO_free(in);
(gdb) p pkey
$1 = (EVP_PKEY *) 0x7ea480
(gdb) f
#0 ossl_ec_key_initialize (argc=2, argv=0x7fffe9eff048, self=140737044094080)
at ../../../../ext/openssl/ossl_pkey_ec.c:167
167 BIO_free(in);
So, in the code below, after ossl_pkey_read_generic fails to get the pkey, then we want to call the ec_key_new_from_group in FIPS case? Because when ossl_pkey_read_generic fails to get the pkey, we can know the key is not FIPS-approved?
openssl/ext/openssl/ossl_pkey_ec.c
Lines 166 to 172 in 7c34a43
By the way, I was discussing how we can know a key or key name is allowed or not allowed in OpenSSL FIPS at openssl/openssl#21830. Unfortunately I don't find a clear way to know it. But I think if ossl_pkey_read_generic doesn't return the value for pkey, we can assume that the key is not allowed in OpenSSL FIPS.
The "-----BEGIN EC PRIVATE KEY-----" PEM can't be decoded in FIPS 140-compliant systems because it uses MD5 to derive encryption keys from passwords (#643, #645). I don't think AES-128-CBC is the issue here.
The error message is unfortunate. As you analyzed it, unlike other PKey types, OpenSSL::PKey::EC.new(string) attempts to treat the given string as a curve name if it turns out not to be a PEM encoding, but the string starting with "-----BEGIN EC PRIVATE KEY-----" is not a curve name either.
The test case (and also corresponding ones in other test_pkey_*.rb files) should be skipped.