Post: How to add columns with default values to really large tables

Author: Sai Wong

_posts/2015-10-16-add-columns-with-default-values-to-large-tables-in-rails-postgres.markdown

@@ -0,0 +1,150 @@
+---
+layout:       post
+title:        Rails Migration - How to add columns with default values to really large tables in Postgres + Rails
+author:       Sai Wong
+summary:
+image:        http://res.cloudinary.com/wework/image/upload/s--xpIlilub--/c_scale,q_jpegmini:1,w_1000/v1443207604/engineering/shutterstock_294201896.jpg
+categories:   data
+---
+
+We had a fairly simple task of adding a couple of columns to a table for our
+Rails app. This is normally a straight forward operation and a boring task at
+best but for us, the fun only just started. The table in question was a fairly
+large table with lots of reads on it and in the spirit of no down time, this
+is the adventure we had.
+
+# The Task
+- Add two columns to the notifications table
+- Both columns have default values
+- Table has 2.2 MM rows!
+
+# Attempt #1
+```ruby
+class AddPhoneFlagsToNotifications < ActiveRecord::Migration
+  def change
+    add_column :notifications, :text_message, :boolean, default: false
+    add_column :notifications, :call_phone, :boolean, default: false
+  end
+end
+```
+
+## Problem
+- Migration takes hours!
+- The notifications table is locked
+- Entire application grinds to a halt
+
+## Reason
+- Column creation with default values causes all rows to be touched at the same time
+- Updates are a slow operation in Postgres since it has to guarantee consistency
+- That guarantee results in whole table locking
+
+## Solution
+- Postgres can create null columns extremely fast! Even on a huge table!
+- We can split the work to two tasks, creating the columns and populating the default value
+
+# Attempt #2
+
+```ruby
+class AddPhoneFlagsToNotifications < ActiveRecord::Migration
+  def change
+    add_column :notifications, :text_message, :boolean
+    add_column :notifications, :call_phone, :boolean
+
+    execute <<-SQL
+      ALTER TABLE notifications
+        ALTER COLUMN text_message SET DEFAULT false,
+        ALTER COLUMN call_phone SET DEFAULT false
+    SQL
+
+    last_id = Notification.last.id
+    batch_size = 10000
+    (0..last_id).step(batch_size).each do |from_id|
+      to_id = from_id + batch_size
+      execute <<-SQL
+        UPDATE notifications
+          SET
+            text_message = false,
+            call_phone = false
+          WHERE id BETWEEN #{from_id} AND #{to_id}
+      SQL
+    end
+  end
+end
+```
+
+## Problem
+- Migration takes hours!
+- The notifications table is still locked!
+- Entire application grinds to a halt
+
+## Reason
+- Rails migration tasks are always wrapped in a transaction to allow for rollbacks
+- The column adds AND the row updates are in one gigantic transaction!
+- Transactions guarantee consistency
+- That guarantee results in whole table locking again!
+
+## Solution
+- You can disable the transaction handle in Rails migration by calling “disable_ddl_transaction!” in your migration task
+- But you have to handle transactions on your own
+- We can then run each step in its own transaction
+- Add our own error handling to rollback operation
+
+# Attempt #3
+
+```ruby
+class AddPhoneFlagsToNotifications < ActiveRecord::Migration
+  disable_ddl_transaction!
+
+  def up
+    ActiveRecord::Base.transaction do
+      add_column :notifications, :text_message, :boolean, default: nil
+      add_column :notifications, :call_phone, :boolean, default: nil
+
+      sql = <<-SQL
+        ALTER TABLE notifications
+          ALTER COLUMN text_message SET DEFAULT false,
+          ALTER COLUMN call_phone SET DEFAULT false
+      SQL
+      execute(sql)
+    end
+
+
+    last_id = Notification.last.id
+    batch_size = 10000
+    (0..last_id).step(batch_size).each do |from_id|
+      to_id = from_id + batch_size
+      ActiveRecord::Base.transaction do
+        execute <<-SQL
+          UPDATE notifications
+            SET
+              text_message = false,
+              call_phone = false
+            WHERE id BETWEEN #{from_id} AND #{to_id}
+        SQL
+      end
+    end
+
+    rescue => e
+      # roll back our work
+      down
+      raise e
+  end
+end
+```
+
+## Result
+- Migration takes hours!
+- There is no table locking
+- Application is slower due to all the writes to notifications table
+- Nothing grinds to a halt
+
+# Takeaways
+- Always be mindful of the number of rows affected in the migration
+- Be mindful of the transaction size
+- Leverage Postgres features
+
+## Possible alternate solution
+- Handle NULL case in code to treat as the desired default value
+- Add view in database to do mapping for us
+- Add trigger to only update rows that are actively queried
+- ???