undefined method 'to_octet_string' for OpenSSL::PKey::EC

[Chris Liaw]

Hi, I've hit this error undefined method 'to_octet_string' when invoking method dh_compute_key() of ECC key

Step to reproduce

require 'openssl'

k1 = OpenSSL::PKey::EC.generate("prime256v1")
k2 = OpenSSL::PKey::EC.generate("prime256v1")

pk1 = k1.public_to_der
pk2 = k2.public_to_der

rpk1 = OpenSSL::PKey::EC.new(pk1)
rpk2 = OpenSSL::PKey::EC.new(pk2)

s1 = k1.dh_compute_key(k1.public_key)   # ==> No error though
s2 = k1.dh_compute_key(k1.public_key)   # ==> No error though

s1 = k1.dh_compute_key(rpk1)  #==> Error undefined method 'to_octet_string' shall be thrown 

     Failure/Error: s1 = k1.dh_compute_key(rpk1)
     
     NoMethodError:
       undefined method `to_octet_string' for #<OpenSSL::PKey::EC:0x00007f59b8e42f78 oid=id-ecPublicKey>

Environment

My environment as followed:

• OS : Linux Mint 21.1, Kernel 5.15.0-69-generic
• Ruby version : ruby 3.2.1 (2023-02-08 revision 31819e82c8) [x86_64-linux]
• gem list openssl : openssl (default: 3.1.0)
• 3.2.1 :002 > OpenSSL::VERSION => "3.1.0"
• 3.2.1 :003 > OpenSSL::OPENSSL_VERSION => "OpenSSL 3.0.2 15 Mar 2022"

Update

After checking out the source code and read the test cases, the following code success however

require 'openssl'

k1 = OpenSSL::PKey::EC.generate("prime256v1")
k2 = OpenSSL::PKey::EC.generate("prime256v1")

pk1 = k1.public_key.to_bn
pk2 = k2.public_key.to_bn

rpk1 = OpenSSL::PKey::EC::Point.new(OpenSSL::PKey::EC::Group.new("prime256v1"), pk1)
rpk2 = OpenSSL::PKey::EC::Point.new(OpenSSL::PKey::EC::Group.new("prime256v1"), pk2)

s1 = k1.dh_compute_key(rpk2) 
s2 = k2.dh_compute_key(rpk1) 
assert(s1 == s2).to be true

Hope fully this can speed up the bug hunting?

Thanks!

[Kazuki Yamaguchi]

OpenSSL::PKey::EC#dh_compute_key takes an instance of OpenSSL::PKey::EC::Point, which can be obtained by OpenSSL::PKey::EC#public_key.

FWIW:

k1 = OpenSSL::PKey::EC.generate("prime256v1")
rpk2 = OpenSSL::PKey::EC.new(OpenSSL::PKey::EC.generate("prime256v1").public_to_der)
p k1.dh_compute_key(rpk2.public_key)

is also equivalent to

k1 = OpenSSL::PKey::EC.generate("prime256v1")
rpk2 = OpenSSL::PKey::EC.new(OpenSSL::PKey::EC.generate("prime256v1").public_to_der)
p k1.derive(rpk2)