Migrating encrypted database fields when upgrading to Rails 7

Question

Issue

I had to upgrade my RoR app to Rails 7 due to this issue. When making this upgrade, my db columns which were being encrypted with the Lockbox gem were no longer able to be read as Rails was using the native decryption to try and decrypt the fields. I posted about it as an issue on GitHub, but am also wondering if anyone else has a solution for migrating the data out of one encryption format and into the new native encryption that will be shipping with Rails 7.0 (Currently the stable version of Rails is 6.1.4 and Rails 7.0.alpha is on the main branch on GitHub)

Code

app/models/journal_entry.rb

class JournalEntry < ApplicationRecord
  belongs_to :prayer_journal

  encrypts  :content
  validates :content, presence: true  
end

db/schema.rb

create_table "journal_entries", force: :cascade do |t|
    t.bigint "prayer_journal_id", null: false
    t.datetime "created_at", precision: 6, null: false
    t.datetime "updated_at", precision: 6, null: false
    t.text "content_ciphertext"
    t.index ["prayer_journal_id"], name: "index_journal_entries_on_prayer_journal_id"
  end

Console output of the first Journal Entry

#<JournalEntry:0x00007f95364745c8
 id: 1,
 prayer_journal_id: 1,
 created_at: Sat, 15 May 2021 00:00:00.000000000 UTC +00:00,
 updated_at: Sat, 17 Jul 2021 03:12:34.951395000 UTC +00:00,
 content_ciphertext: "l6lfumUqk9RqUHMf0aVUfL2sL+WqkhBmHpyqKqMtxD4=",
 content: nil>

Answer 1

After a few hours pouring over the Rails guides and various blog posts talking about the new native encryption, I was able to figure out how to migrate the data. It is a multi-step process, but I felt that I would place it here for future help to others.

First, I do want to say that it may be possible to list other encryption/decryption providers if I am reading the guides correctly. I was unable to figure that out, and so decided to use what I do know to create a solution.

How I came up with the solution

I noticed that in my schema there wasn't actually a "content" column, but rather a "content_ciphertext" column and when lockbox was being called in encrypt :content, it would encrypt and place it in that column. And I could call JournalEntry.first.content to have it decrypt the content_ciphertext field and provide the plain text. This is why, after upgrading to Rails 7 and the native encryption, it kept saying that the column content was nil; because in actuality, there was no column by that name. Rails 7 uses the exact naming within the schema, not appending 'ciphertext' or the like to the column name.

Having this knowledge solved the rest for me.

Steps to solve

1. BEFORE UPGRADING THE RAILS VERSION: Create a migration to add a content field to the tables with the encrypted data. In my case there where three tables. So, I ran this code: rails g migration AddUnencryptedContentFieldToDatabaseTabels

and changed the migration file to look like this:

# db/migrate/*******_add_unencrypted_content_field_to_database_tabels.rb

class AddUnencryptedContentFieldToDatabaseTabels < ActiveRecord::Migration[6.1]
  def up
    add_column    :journal_entries,        :unencrypted_content,         :text
    add_column    :prayer_requests,        :unencrypted_content,         :text
    add_column    :prayer_request_updates, :unencrypted_content,         :text
  end

  def down
    remove_column :journal_entries,        :unencrypted_content
    remove_column :prayer_requests,        :unencrypted_content
    remove_column :prayer_request_updates, :unencrypted_content
  end
end

That done, I wrote a rake task to go through and copy all the encrypted fields over to an unencrypted column.

# lib/tasks/switch_encryption_1.rake

desc 'This goes through and copies encrypted data to a non-encrypted field to start the process of migrating to new native encryption.'
task :switch_encryption_1 => :environment do
    puts "Journal Entries where content needs to be unencrypted: " + JournalEntry.where(unencrypted_content:nil).count.to_s
    JournalEntry.where(unencrypted_content:nil).each do |j|
        j.update(unencrypted_content:j.content)
    end
    puts "Journal Entries where content needs to be unencrypted after code run: " + JournalEntry.where(unencrypted_content:nil).count.to_s

    puts "Prayer Requests where content needs to be unencrypted: " + PrayerRequest.where(unencrypted_content:nil).count.to_s
    PrayerRequest.where(unencrypted_content:nil).each do |r|
        r.update(unencrypted_content:r.content)
    end
    puts "Prayer Requests where content needs to be unencrypted after code run: " + PrayerRequest.where(unencrypted_content:nil).count.to_s

    puts "Prayer Request Updates where content needs to be unencrypted: " + PrayerRequestUpdate.where(unencrypted_content:nil).count.to_s
    PrayerRequestUpdate.where(unencrypted_content:nil).each do |u|
        u.update(unencrypted_content:u.content)
    end
    puts "Prayer Request Updates where content needs to be unencrypted after code run: " + PrayerRequestUpdate.where(unencrypted_content:nil).count.to_s
end

Those both written, I could now deploy the code to production. Once deployed I would run rake db:migrate in the production console, then rake switch_encryption_1 to go through and decrypt and copy all of the fields to the new column.

I could also then test to make sure that the data is actually copied and decrypted before proceeding.

2. Back in development, I can now update my Gemfile the new Rails main branch as I have decrypted the fields. So, I change the Gemfile to this:

   gem 'rails', :github => 'rails/rails', :branch => 'main'

You will then need to create the encryption keys by running bin/rails db:encryption:init in the console and copying the values to the credentials file. If you don't know how to do that, you run this code EDITOR=nano rails credentials:edit and copy the values into that file:

active_record_encryption:
  primary_key: xxxxxxxxxxxxxxxxxxx
  deterministic_key: xxxxxxxxxxxxxxxxxxx
  key_derivation_salt: xxxxxxxxxxxxxxxxxxx

Then follow the prompts to save and exit. For me that is Control + the capital letter 'O' to Write Out and then Control + the capital letter 'X' to exit. This will work for development. Since Rails 6, we have been able to set different credentials for the different environments. So, you'd copy the same data, but in the console run EDITOR=nano rails credentials:edit --environment production to get to the production credentials. (REMEMBER TO KEEP THESE KEYS VERY SAFE AND TO NOT CHECK THEM INTO VERSION CONROL)

Then I created another migration rails g migration AddContentFieldToDatabaseTabels

and changed the migration file to look like this:

# db/migrate/*******_add_content_field_to_database_tabels.rb

class AddContentFieldToDatabaseTabels < ActiveRecord::Migration[6.1]
  def up
    add_column    :journal_entries,        :content,             :text
    add_column    :prayer_requests,        :content,             :text
    add_column    :prayer_request_updates, :content,             :text

    remove_column :journal_entries,        :content_ciphertext
    remove_column :prayer_requests,        :content_ciphertext
    remove_column :prayer_request_updates, :content_ciphertext
  end

  def down
    remove_column :journal_entries,        :content
    remove_column :prayer_requests,        :content
    remove_column :prayer_request_updates, :content

    add_column    :journal_entries,        :content_ciphertext,  :text
    add_column    :prayer_requests,        :content_ciphertext,  :text
    add_column    :prayer_request_updates, :content_ciphertext,  :text
  end
end

You'll probably notice that I also added in code to remove the old encrypted column. This is because that will no longer be used and I have already verified that the content is now saved in the unencrypted_content columns.

I then wrote another rake task to go through and copy all of the data from the unencrypted_content columns to the content columns. And since my models already have the code encrypts :content from before with the Lockbox gem, I don't need to add that to the models to let Rails know to encrypt those columns.

# lib/tasks/switch_encryption_2.rake

desc 'This goes through and encrypts the unencrypted data and copies it to the encrypted field to finish migrating to new native encryption.'
task :switch_encryption_2 => :environment do
    JournalEntry.all.each do |j|
        j.update(content:j.unencrypted_content)
    end

    PrayerRequest.all.each do |r|
        r.update(content:r.unencrypted_content)
    end

    PrayerRequestUpdate.all.each do |u|
        u.update(content:u.unencrypted_content)
    end

    puts "Finished Encrypting"
end

Now, deploy. Your production credentials should also have been deployed for encryption. Now run this in the production console: rake db:migrate and rake switch_encryption_2. After I did that, I verified that the encryption worked.

3. I can now just create another migration in development to delete the unencrypted table columns. Like so: rails g migration DeleteUnencryptedContentFieldFromDatabaseTables

db/migrate/*******_delete_unencrypted_content_field_to_database_tabels.rb

class DeleteUnencryptedContentFieldToDatabaseTabels < ActiveRecord::Migration[6.1]
    def up
        remove_column :journal_entries,        :unencrypted_content
        remove_column :prayer_requests,        :unencrypted_content
        remove_column :prayer_request_updates, :unencrypted_content
    end

    def down
        add_column    :journal_entries,        :unencrypted_content,  :text
        add_column    :prayer_requests,        :unencrypted_content,  :text 
        add_column    :prayer_request_updates, :unencrypted_content,  :text
    end
end

Push that to production and run rake db:migrate.

At this point, everything should be migrated to the new native Rails 7 encryption.

I hope that this helps future coders. Happy Coding!

BONUS SECTION

For those paranoid among us, or working with very sensitive data and needing to make sure that the unencrypted columns are no more. Here is a third rake task that I created that goes through and writes over the columns with nil. You can run this before deploying the migration to delete the columns. But, really, this is probably just overkill:

desc 'After verifying that the data is now encrypted and able to be decrypted, this task will go through and erase the unencrypted fields'
task :switch_encryption_3 => :environment do
    JournalEntry.all.each do |j|
        j.update(unencrypted_content:nil)
    end

    PrayerRequest.all.each do |r|
        r.update(unencrypted_content:nil)
    end

    PrayerRequestUpdate.all.each do |u|
        u.update(unencrypted_content:nil)
    end

    puts "Finished Enrasing Unencrypted Data. You will need to run a new migration to delete the 'unencrypted_content' fields."
end