How do I retrieve a list of created IDs for bulk insert in Active Record?

Question

I have three models:

class Coupon < ActiveRecord::Base
  belongs_to :event
  has_many :coupon_events, :dependent => :destroy
  has_many :events, :through => :coupon_events
end 

class Event < ActiveRecord::Base
  belongs_to :event
  has_many :coupon_events, :dependent => :destroy
  has_many :coupons, :through => :coupon_events
end

class CouponEvent < ActiveRecord::Base
  belongs_to :coupon
  belongs_to :event
end

I read through a CSV file to create coupons and coupon_events. This is terribly inefficient since the records are created one at a time and result in multiple queries each including the two insert statements.

I'd like to use a single insert query like this:

coupon_string = " ('abc','AAA'), ('123','BBB')"
Coupon.connection.insert("INSERT INTO coupons (code, name) VALUES"+coupon_string)

I then need to create the second insert query for the CouponEvent model, but I need a list of the returned coupon_ids. Is there a built in method to retrieve the IDs at the time of the insert?

Answer 1

At the moment, the best (but not ideal) solution is to bulk import using "activerecord-import". Unfortunately, that gem does not return the inserted ids, so you'd have to turn around and query to get the ids. That is, you'd bulk insert the Events models, query the db to get them all back in memory. Now you have the Event ids, so you can create the Coupons and bulk insert them. Rinse lather repeat for CouponEvents.

Compared to one round trip per Event, Coupon and CouponEvent - probably thousands of round trips for a file with thousands of rows - you are only doing 2 round trips per model - One to insert the Event, one to fetch the Events back with the ids, ditto Coupon and CouponEvent - total 6 round trips.

Answer 2

Actually I'm not sure if this colud work (if it creates one insert query), but you can try to use #create method with array of parameters:

new_coupons = Coupon.create([
  { :code => "abc", :name => "AAA" },
  { :code => "123", :name => "BBB" }
])

CouponEvent.create([
  { :enevt_id => ..., coupon_id: ...},
  ...
])

To create parameters list for CouponEvent, you neet map returned collection of new_coupons to id's and add event_id's based on coupon codes/names (depends haw it's stored in CVS file).

UPDATE:

I checked by myself, and if first solution doesn't work (I don't simple have models without uniqueness constraints in my code, so I haven't checked), and you use PostgreSQL, you can always do something like this:

res = Coupon.connection.execute(<<-EOSQL)
  INSERT INTO coupons (code, name)
  VALUES #{values}
  RETURNING id, code
EOSQL

You need that last "Returning" clause, so you can fetch id's inserted along with code of inserted row. The you need to map resultset:

res.map {|row|
  { :coupon_id => row["id"],
    :event_id => events.find { |e| e.coupon_code == row["code"] }
  }
}

There is no standard way in SQL to return columns of inserted rows, "RETURNING" clause works in PostgreSQL only, so if you use different database, you need check documentation or insert rows one by one.

You can't also use connection.insert, as in ActiveRecord it returns only id of one inserted row, instead of all rows.

Answer 3

The way to do this is to insert the records with a unique import_id value. The steps are:

1. Add an import_id column to the table. Could be INT or VARCHAR depending on how you generate random IDs.

2. Before the first INSERT, generate a random ID.

3. Do the first multi-value INSERT, using the same import_id for each row.

4. SELECT id FROM first_table WHERE import_id=<the random import ID>

5. Generate second multi-value INSERT using the returned IDs.