Getting big decimals back from a yaml-serialized field in the database with Ruby on Rails

Question

Using Ruby on Rails I have a couple of fields that are serialized (arrays or hashes mostly). Some of those contain BigDecimals. It is very important that those big decimals remain big decimals, but Rails is turning them into floats. How do I get BigDecimals back?

Looking into this issue I found that serializing a big decimal in plain Ruby, without Rails, works as expected:

BigDecimal.new("42.42").to_yaml
 => "--- !ruby/object:BigDecimal 18:0.4242E2\n...\n"

but in a Rails console it doesn't:

BigDecimal.new("42.42").to_yaml
 => "--- 42.42\n"

That number is the string representation of the big decimal, so it's all-right. But when I read it back it is read as a float, so even if I convert it to BigDecimal (something I don't want to do as it's error prone), it is possible I'll lose precision, which isn't acceptable for my app.

I tracked down the culprit to activesupport-3.2.11/lib/active_support/core_ext/big_decimal/conversions.rb which overrides the following method in BigDecimal:

YAML_TAG = 'tag:yaml.org,2002:float'
YAML_MAPPING = { 'Infinity' => '.Inf', '-Infinity' => '-.Inf', 'NaN' => '.NaN' }

# This emits the number without any scientific notation.
# This is better than self.to_f.to_s since it doesn't lose precision.
#
# Note that reconstituting YAML floats to native floats may lose precision.
def to_yaml(opts = {})
  return super if defined?(YAML::ENGINE) && !YAML::ENGINE.syck?

  YAML.quick_emit(nil, opts) do |out|
    string = to_s
    out.scalar(YAML_TAG, YAML_MAPPING[string] || string, :plain)
  end
end

Why would they do that? And more importantly, how do I work-around it?

Answer 1

The ActiveSupport core extension code you mentioned is "already" fixed in the master branch (the commit is about an year old and undoes an implementation as old as Rails 2.1.0), but since Rails 3.2 only gets security updates, your app might be stuck with the old implementation.

I think you'll have three options:

1. Port your Rails app to Rails 4.
2. Backport Psych's BigDecimal#to_yaml implementation (monkey patch the monkey patch).
3. Switch to Syck as YAML engine.

Each option has its own disadvantage:

Porting to Rails 4 seems to me the best alternative, if you have the time to do so (the commit mentioned above is available in Rails since v4.0.0.beta1). Since it's not released yet, you'd have to work with the beta. I don't suspect any grand changes to come, though some GSoC ideas read as if they still could make it into the 4.0 release...

Monkey patching the ActiveSupport monkey patch should be fairly less complex. Though I did not find the original implementation of BigDecimal#to_yaml, a somewhat related question led to this commit. I guess I'll leave it to you (or other StackOverflow users) how to backport that particular method.

As quick'n'dirty workaround, you could simply use Syck as YAML engine. In the same question, user rampion posted this piece of code (which you could place in an initializer file):

YAML::ENGINE.yamler = 'syck'

class BigDecimal
  def to_yaml(opts={})
    YAML::quick_emit(object_id, opts) do |out|
      out.scalar("tag:induktiv.at,2007:BigDecimal", self.to_s)
    end
  end
end

YAML.add_domain_type("induktiv.at,2007", "BigDecimal") do |type, val|
  BigDecimal.new(val)
end

The main disadvantage here (besides the unavailability of Syck on Ruby 2.0.0) is, that you can't read normal BigDecimal dumps within your Rails context, and everybody who want's to read your YAML dumps, needs the same kind of loader:

BigDecimal.new('43.21').to_yaml
#=> "--- !induktiv.at,2007/BigDecimal 43.21\n"

(Changing the tag to "tag:ruby/object:BigDecimal" won't help either, since it yields !ruby/object/BigDecimal...)

---

Update – things I learned so far

1. The odd behaviour seems to date back to times of Rails 1.2 (you might also say Feb'2007), as per this blog entry.

2. Modifying the config/application.rb in this way did not help:

   require File.expand_path('../boot', __FILE__)
   
   # (a)
   
   %w[yaml psych bigdecimal].each {|lib| require lib }
   class BigDecimal
     # backup old method definitions
     @@old_to_yaml = instance_method :to_yaml
     @@old_to_s    = instance_method :to_s
   end
   
   require 'rails/all'
   
   # (b)
   
   class BigDecimal
     # restore the old behavior
     define_method :to_yaml do |opts={}|
       @@old_to_yaml.bind(self).(opts)
     end
     define_method :to_s do |format='E'|
       @@old_to_s.bind(self).(format)
     end
   end
   
   # (c)
   

   At different points (here a, b and c) a BigDecimal.new("42.21").to_yaml has yielded some interesting output:

   # (a) => "--- !ruby/object:BigDecimal 18:0.4221E2\n...\n"
   # (b) => "--- 42.21\n...\n"
   # (c) => "--- 0.4221E2\n...\n"
   

   where a is the default behaviour, b is caused by the ActiveSupport Core Extension and c should have been the same result as a. Maybe I'm missing something...

3. On carefully rereading you question, I had this idea: Why not serialize in another format, like, JSON? Add another column to your database and migrate over time like this:

   class Person < ActiveRecord::Base
     # the old serialized field
     serialize :preferences
   
     # the new one. once fully migrated, drop old preferences column
     # rename this to preferences and remove the getter/setter methods below
     serialize :pref_migration, JSON
   
     def preferences
       if pref_migration.blank?
         pref_migration = super
         save! # maybe don't use bang here
       end
       pref_migration
     end
   
     def preferences=(*data)
       pref_migration = *data
     end
   end
   

Answer 2

In case you are using Rails 4.0 or above (but below 4.2), you can work-around it by removing the method BigDecimal#encode_with.

You can archive that by using undef_method:

require 'bigdecimal'
require 'active_support/core_ext/big_decimal'

class BigDecimal
  undef_method :encode_with
end

I put this code inside an initializer and now it works. This "revert" of Rails monkey patch won't be necessary in Rails 4.2, since this commit removes the monkey patch.

Answer 3

For rails 3.2, the following works:

# config/initializers/backport_yaml_bigdecimal.rb

require "bigdecimal"
require "active_support/core_ext/big_decimal"

class BigDecimal
  remove_method :encode_with
  remove_method :to_yaml
end

Without this patch, in a rails 3.2 console:

irb> "0.3".to_d.to_yaml
=> "--- 0.3\n...\n"

With this patch:

irb> "0.3".to_d.to_yaml
=> "--- !ruby/object:BigDecimal 18:0.3E0\n...\n"

You might like to wrap this in a version test with documentation and deprecation warnings, something like:

# BigDecimals should be correctly tagged and encoded in YAML as ruby objects
# instead of being cast to/from floating point representation which may lose
# precision.
#
# This is already upstream in Rails 4.2, so this is a backport for now.
#
# See http://stackoverflow.com/questions/16031850/getting-big-decimals-back-from-a-yaml-serialized-field-in-the-database-with-ruby
#
# Without this patch:
#
#   irb> "0.3".to_d.to_yaml
#   => "--- 0.3\n...\n"
#
# With this patch:
#
#   irb> "0.3".to_d.to_yaml
#   => "--- !ruby/object:BigDecimal 18:0.3E0\n...\n"
#
if Gem::Version.new(Rails.version) < Gem::Version.new("4.2")
  require "bigdecimal"
  require "active_support/core_ext/big_decimal"

  class BigDecimal
    # Rails 4.0.0 removed #to_yaml
    # https://github.com/rails/rails/commit/d8ed247c7f11b1ca4756134e145d2ec3bfeb8eaf
    if Gem::Version.new(Rails.version) < Gem::Version.new("4")
      remove_method :to_yaml
    else
      ActiveSupport::Deprecation.warn "Hey, you can remove this part of the backport!"
    end

    # Rails 4.2.0 removed #encode_with
    # https://github.com/rails/rails/commit/98ea19925d6db642731741c3b91bd085fac92241
    remove_method :encode_with
  end
else
  ActiveSupport::Deprecation.warn "Hey, you can remove this backport!"
end