Forced encoding for deterministic encryption and other improvements (rails#42491)

* Fix: re-encrypting will preserve case when ignore_case is used

The implementation was outdated in relation to the new approach where we
override accessors.

* Add support for exists?(...) when querying encrypted attributes

* Add a new option to force encoding for deterministic attributes

Different encodings result in different ciphertexts. For
deterministically encrypted attributes, this can result in having
attributes with the same value that fails to satisfy uniqueness
constraints due to having different encodings.

This adds a new option `forced_encoding_for_deterministic_encryption:`
that will be UTF-8 by default. User can disabled this new behavior by
setting the option to nil.

* Add changelog entry

Author: jorgemanrubia

activerecord/CHANGELOG.md

@@ -1,3 +1,13 @@
+*   Active Record Encryption will now encode values as UTF-8 when using deterministic
+    encryption. The encoding is part of the encrypted payload, so different encodings for
+    different values result in different ciphertexts. This can break unique constraints and
+    queries.
+
+    The new behavior is configurable via `active_record.encryption.forced_encoding_for_deterministic_encryption`
+    that is `Encoding::UTF_8` by default. It can be disabled by setting it to `nil`.
+
+    *Jorge Manrubia*
+
 *   Disable automatic write protection on replicas.
 
     Write protection is no longer automatically enabled for replicas. Write protection should be enabled by the database user settings.

activerecord/lib/active_record/encryption/config.rb

@@ -6,7 +6,7 @@ module Encryption
     class Config
       attr_accessor :primary_key, :deterministic_key, :store_key_references, :key_derivation_salt,
                     :support_unencrypted_data, :encrypt_fixtures, :validate_column_size, :add_to_filter_parameters,
-                    :excluded_from_filter_parameters, :extend_queries, :previous_schemes
+                    :excluded_from_filter_parameters, :extend_queries, :previous_schemes, :forced_encoding_for_deterministic_encryption
 
       def initialize
         set_defaults
@@ -30,6 +30,7 @@ def set_defaults
           self.add_to_filter_parameters = true
           self.excluded_from_filter_parameters = []
           self.previous_schemes = []
+          self.forced_encoding_for_deterministic_encryption = Encoding::UTF_8
 
           # TODO: Setting to false for now as the implementation is a bit experimental
           self.extend_queries = false

activerecord/lib/active_record/encryption/encryptable_record.rb

@@ -176,11 +176,7 @@ def has_encrypted_attributes?
 
         def build_encrypt_attribute_assignments
           Array(self.class.encrypted_attributes).index_with do |attribute_name|
-            if source_attribute_name = self.class.source_attribute_from_preserved_attribute(attribute_name)
-              self[source_attribute_name]
-            else
-              self[attribute_name]
-            end
+            self[attribute_name]
           end
         end
 

activerecord/lib/active_record/encryption/encrypted_attribute_type.rb

@@ -31,6 +31,8 @@ def deserialize(value)
       end
 
       def serialize(value)
+        value = force_encoding_if_needed(value)
+
         if serialize_with_oldest?
           serialize_with_oldest(value)
         else
@@ -49,12 +51,24 @@ def previous_types # :nodoc:
       end
 
       private
+        def force_encoding_if_needed(value)
+          if deterministic? && forced_encoding_for_deterministic_encryption && value && value.encoding != forced_encoding_for_deterministic_encryption
+            value.encode(forced_encoding_for_deterministic_encryption, invalid: :replace, undef: :replace)
+          else
+            value
+          end
+        end
+
+        def forced_encoding_for_deterministic_encryption
+          ActiveRecord::Encryption.config.forced_encoding_for_deterministic_encryption
+        end
+
         def previous_schemes_including_clean_text
           previous_schemes.including((clean_text_scheme if support_unencrypted_data?)).compact
         end
 
         def previous_types_without_clean_text
-          @previous_types_without_clean_text  ||= build_previous_types_for(previous_schemes)
+          @previous_types_without_clean_text ||= build_previous_types_for(previous_schemes)
         end
 
         def build_previous_types_for(schemes)

activerecord/lib/active_record/encryption/extended_deterministic_queries.rb

@@ -82,7 +82,12 @@ module RelationQueries
         include EncryptedQueryArgumentProcessor
 
         def where(*args)
-          process_encrypted_query_arguments(args, true) unless self.deterministic_encrypted_attributes&.empty?
+          process_encrypted_query_arguments_if_needed(args)
+          super
+        end
+
+        def exists?(*args)
+          process_encrypted_query_arguments_if_needed(args)
           super
         end
 
@@ -93,6 +98,11 @@ def find_or_create_by(attributes, &block)
         def find_or_create_by!(attributes, &block)
           find_by(attributes.dup) || create!(attributes, &block)
         end
+
+        private
+          def process_encrypted_query_arguments_if_needed(args)
+            process_encrypted_query_arguments(args, true) unless self.deterministic_encrypted_attributes&.empty?
+          end
       end
 
       module CoreQueries

activerecord/test/cases/encryption/encryptable_record_api_test.rb

@@ -91,6 +91,22 @@ class ActiveRecord::Encryption::EncryptableRecordApiTest < ActiveRecord::Encrypt
     assert_equal encoding, post.reload.title.encoding
   end
 
+  test "encrypt will honor forced encoding for deterministic attributes" do
+    ActiveRecord::Encryption.config.forced_encoding_for_deterministic_encryption = Encoding::UTF_8
+
+    book = ActiveRecord::Encryption.without_encryption { EncryptedBook.create!(name: "Dune".encode("US-ASCII")) }
+    book.encrypt
+    assert Encoding::UTF_8, book.reload.name.encoding
+  end
+
+  test "encrypt won't force encoding for deterministic attributes when option is nil" do
+    ActiveRecord::Encryption.config.forced_encoding_for_deterministic_encryption = nil
+
+    book = ActiveRecord::Encryption.without_encryption { EncryptedBook.create!(name: "Dune".encode("US-ASCII")) }
+    book.encrypt
+    assert Encoding::US_ASCII, book.reload.name.encoding
+  end
+
   test "encrypt will preserve case when :ignore_case option is used" do
     ActiveRecord::Encryption.config.support_unencrypted_data = true
 
@@ -101,7 +117,20 @@ class ActiveRecord::Encryption::EncryptableRecordApiTest < ActiveRecord::Encrypt
 
     book.encrypt
 
+    assert_equal "Dune", book.reload.name
+  end
+
+  test "re-encrypting will preserve case when :ignore_case option is used" do
+    ActiveRecord::Encryption.config.support_unencrypted_data = true
+
+    book = create_unencrypted_book_ignoring_case name: "Dune"
+
+    ActiveRecord::Encryption.without_encryption { assert_equal "Dune", book.reload.name }
     assert_equal "Dune", book.name
+
+    2.times { book.encrypt }
+
+    assert_equal "Dune", book.reload.name
   end
 
   test "encrypt attributes encrypted with a previous encryption scheme" do

activerecord/test/cases/encryption/encryptable_record_test.rb

@@ -262,6 +262,30 @@ def name
     assert book.name_previously_changed?
   end
 
+  test "forces UTF-8 encoding for deterministic attributes by default" do
+    book = EncryptedBook.create!(name: "Dune".encode("ASCII-8BIT"))
+    assert_equal Encoding::UTF_8, book.reload.name.encoding
+  end
+
+  test "forces encoding for deterministic attributes based on the configured option" do
+    ActiveRecord::Encryption.config.forced_encoding_for_deterministic_encryption = Encoding::US_ASCII
+
+    book = EncryptedBook.create!(name: "Dune".encode("ASCII-8BIT"))
+    assert_equal Encoding::US_ASCII, book.reload.name.encoding
+  end
+
+  test "forced encoding for deterministic attributes will replace invalid characters" do
+    book = EncryptedBook.create!(name: "Hello \x93\xfa".b)
+    assert_equal "Hello ��", book.reload.name
+  end
+
+  test "forced encoding for deterministic attributes can be disabled" do
+    ActiveRecord::Encryption.config.forced_encoding_for_deterministic_encryption = nil
+
+    book = EncryptedBook.create!(name: "Dune".encode("US-ASCII"))
+    assert_equal Encoding::US_ASCII, book.reload.name.encoding
+  end
+
   private
     class FailingKeyProvider
       def decryption_key(message) end

activerecord/test/cases/encryption/extended_deterministic_queries_test.rb

@@ -32,4 +32,9 @@ class ActiveRecord::Encryption::ExtendedDeterministicQueriesTest < ActiveRecord:
     EncryptedBook.find_or_create_by!(name: "Dune")
     assert EncryptedBook.find_by(name: "Dune")
   end
+
+  test "exists?(...) works" do
+    ActiveRecord::Encryption.without_encryption { EncryptedBook.create! name: "Dune" }
+    assert EncryptedBook.exists?(name: "Dune")
+  end
 end

activerecord/test/cases/encryption/helper.rb

@@ -144,7 +144,7 @@ class ActiveRecord::EncryptionTestCase < ActiveRecord::TestCase
   # , PerformanceHelpers
 
   ENCRYPTION_ATTRIBUTES_TO_RESET = %i[ primary_key deterministic_key key_derivation_salt store_key_references
-    key_derivation_salt support_unencrypted_data encrypt_fixtures ]
+    key_derivation_salt support_unencrypted_data encrypt_fixtures forced_encoding_for_deterministic_encryption ]
 
   setup do
     ENCRYPTION_ATTRIBUTES_TO_RESET.each do |property|

guides/source/active_record_encryption.md

@@ -240,6 +240,24 @@ config.active_record.encryption.add_to_filter_parameters = false
 ```
 In case you want exclude specific columns from this automatic filtering, add them to `config.active_record.encryption.excluded_from_filter_parameters`.
 
+### Encoding
+
+The library will preserve the encoding for string values encrypted non-deterministically. 
+
+For values encrypted deterministically, by default, the library will force UTF-8 encoding. The reason is that encoding is stored along with the encrypted payload. This means that the same value with a different encoding will result in different ciphertexts when encrypted. You normally want to avoid this to keep queries and uniqueness constraints working, so the library will perform the conversion automatically on your behalf.
+
+You can configure the desired default encoding for deterministic encryption with:
+
+```ruby
+config.active_record.encryption.forced_encoding_for_deterministic_encryption = Encoding::US_ASCII
+```
+
+And you can disable this behavior and preserve the encoding in all cases with:
+
+```ruby
+config.active_record.encryption.forced_encoding_for_deterministic_encryption = nil
+```
+
 ## Key management
 
 Key management strategies are implemented by key providers. You can configure key providers globally or on a per-attribute basis.
@@ -404,6 +422,7 @@ The available config options are:
 | `primary_key`                                                 | The key or lists of keys that is used to derive root data-encryption keys. They way they are used depends on the key provider configured. It's preferred to configure it via a credential `active_record_encryption.primary_key`. |
 | `deterministic_key`                                          | The key or list of keys used for deterministic encryption. It's preferred to configure it via a credential `active_record_encryption.deterministic_key`. |
 | `key_derivation_salt`                                        | The salt used when deriving keys. It's preferred to configure it via a credential `active_record_encryption.key_derivation_salt`. |
+| `forced_encoding_for_deterministic_encryption` | The default encoding for attributes encrypted deterministically. You can disable forced encoding by setting this option to `nil`. It's `Encoding::UTF_8` by default. |
 
 NOTE: It's recommended to use Rails built-in credentials support to store keys. If you prefer to set them manually via config properties, make sure you don't commit them with your code (e.g: use environment variables).