Extra calls to Active Record encryption key provider

[Andrew Kane]

Steps to reproduce

Hi, it looks like the key provider gets called a number of times when creating a record with a single encrypted attribute. This can cause extra latency and cost if you use an external key management service like Amazon KMS (discovered when working on ankane/kms_encrypted#33).

# frozen_string_literal: true

require "bundler/inline"

gemfile(true) do
  source "https://rubygems.org"

  git_source(:github) { |repo| "https://github.com/#{repo}.git" }

  gem "rails", github: "rails/rails", branch: "main"
  gem "sqlite3"
end

require "active_record"
require "minitest/autorun"
require "logger"

# This connection will do for database-independent bug reports.
ActiveRecord::Base.establish_connection(adapter: "sqlite3", database: ":memory:")
ActiveRecord::Base.logger = Logger.new(STDOUT)

ActiveRecord::Schema.define do
  create_table :users, force: true do |t|
    t.text :email
  end
end

class User < ActiveRecord::Base
  encrypts :email
end

$encryption_calls = 0
$decryption_calls = 0

class MyKeyProvider
  def encryption_key
    puts "encryption call"
    $encryption_calls += 1
    ActiveRecord::Encryption::Key.new("0"*32)
  end

  def decryption_keys(encrypted_message)
    puts "decryption call"
    $decryption_calls += 1
    [ActiveRecord::Encryption::Key.new("0"*32)]
  end
end

ActiveRecord::Encryption.configure(
  key_provider: MyKeyProvider.new,
  primary_key: "secret",
  key_derivation_salt: "salt",
  deterministic_key: nil
)

class BugTest < Minitest::Test
  def test_create
    User.create!(email: "[e-mail]")
    assert_equal 1, $encryption_calls
    assert_equal 0, $decryption_calls
  end
end

Expected behavior

1 call to encryption_key and no calls to decryption_keys

Actual behavior

3 calls to encryption_key and 1 call to decryption_keys

System configuration

Rails version: main

Ruby version: 3.0.0

[Alex Ghiculescu]

@jorgemanrubia can you confirm if this is expected or a bug? It feels like a bug but it's a pretty new feature (🎉 ) and I haven't used it yet so wouldn't mind a sanity check.

[Jorge Manrubia]

Definitely not intended. Using an external key provider looks like a good reason to optimize this. I'll have a look and see how to fix. Thanks for the great report @ankane.

[Jorge Manrubia]

@ankane I had a look and the problem with the 3 encryption_key calls and the problems happens in the Active Record attribute internals, not in the encryption system itself. Essentially, it serializes (encrypts) attributes:

1. to detect if there are changes
2. to write to the database
3. to clear assignments (some internal logic I am not sure what is about)

I am afraid that supporting this properly would imply some major changes to the way Active Record encryption is designed. As you say in this thread, there is no way of accessing the record from the key provider. This limitation is partially motivated because in Rails attribute types doesn't really know about the record or attribute they serialize or deserialize. I can see how this information could be useful but, again, this would imply a major change in how everything is wired up.

Maybe an acceptable workaround would be to throttle the generation of new KMS keys to one every 10ms or whatever threshold makes sense.So that you cache and return the last KEY for requests within a given "X"ms slot.

And it's the same for decrypting. Rails deserialize existing values in the database to detect changes, when writing to the database. I'd suggest creating some size-limited hash-based cache, so that you avoid unnecessary calls to Amazon KMS.

[Andrew Kane]

Hey @jorgemanrubia, thanks for looking into it! Will close this out.