Table name corruption errors in ActiveRecord

Question

Sporadically we get PG::UndefinedTable errors while using ActiveRecord. The association table name is some how corrupted and I quite often see Cancelled appended to the end of the table name.

E.g:

ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR:  relation "fooCancell" does not exist 
ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR:  relation "Cancelled" does not exist
ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR:  relation "barC" does not exist

In the example above, I have obfuscated the table name by using foo and bar.

We see this errors when the rails project is running inside Puma. Queue workers seems to be doing okay.

The tables in the error message doesn't correspond to real tables or models. It looks like the case of memory corruption. Has anyone seen such issues? If so how did you get around it?

puma.rb

on_worker_boot do
  ActiveRecord::Base.establish_connection
end

database.yml

production:
  url:  <%= ENV["DATABASE_URL"] %>
  pool: <%= ENV['DB_CONNECTION_POOL_SIZE'] || 5%>
  reaping_frequency: <%= ENV['DB_CONNECTION_REAPING_FREQUENCY'] || 10 %>
  prepared_statements: false

Answer 1

I'm hazarding a guess here, based on this possibly related error...

But you might be either:

1. calling fork within your application; OR
2. calling ActiveRecord routines (using database calls) before the server (puma) is forking it's worker processes (during the app initialization).

Either of these will break ActiveRecord's synchronization and cause multiple processes to share the database connection pool without synchronizing it's use (resulting in interlaced and corrupt database commands).

If you are using fork, make sure to close all the ActiveRecord database connections and reinitialize the connection pool (there's a function call that does it, but I don't remember it of the top of my head, maybe ActiveRecord.disconnect! or ActiveRecord.connection_pool.disconnect!).

Otherwise, before running Puma (either during the initialization process or using Puma's after_fork), close all the ActiveRecord database connections and reinitialize the connection pool.