Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Bulk inserts taking longer than expected using Dapper

After reading this article I decided to take a closer look at the way I was using Dapper.

I ran this code on an empty database

var members = new List<Member>(); for (int i = 0; i < 50000; i++) {     members.Add(new Member()     {         Username = i.toString(),         IsActive = true     }); }  using (var scope = new TransactionScope()) {     connection.Execute(@" insert Member(Username, IsActive) values(@Username, @IsActive)", members);      scope.Complete(); } 

it took about 20 seconds. That's 2500 inserts/second. Not bad, but not great either considering the blog was achieving 45k inserts/second. Is there a more efficient way to do this in Dapper?

Also, as a side note, running this code through the Visual Studio debugger took over 3 minutes! I figured the debugger would slow it down a little, but I was really surprised to see that much.

UPDATE

So this

using (var scope = new TransactionScope()) {     connection.Execute(@" insert Member(Username, IsActive) values(@Username, @IsActive)", members);      scope.Complete(); } 

and this

    connection.Execute(@" insert Member(Username, IsActive) values(@Username, @IsActive)", members); 

both took 20 seconds.

But this took 4 seconds!

SqlTransaction trans = connection.BeginTransaction();  connection.Execute(@" insert Member(Username, IsActive) values(@Username, @IsActive)", members, transaction: trans);  trans.Commit(); 
like image 878
kenwarner Avatar asked May 21 '12 17:05

kenwarner


People also ask

Does dapper support bulk insert?

Dapper provides the Execute method for inserting data. However, a database roundtrip is required for every data you insert. It's a perfect solution when you need to insert only 1 data but become very fast inefficient as soon as you insert multiple data.

Why bulk insert is faster?

In case of BULK INSERT, only extent allocations are logged instead of the actual data being inserted. This will provide much better performance than INSERT. The actual advantage, is to reduce the amount of data being logged in the transaction log.

Is bulk insert a transaction?

BulkInsert doesn't create a transaction by default. If you want to save multiple lists, you will need to handle the transaction in your code.


2 Answers

The best I was able to achieve was 50k records in 4 seconds using this approach

SqlTransaction trans = connection.BeginTransaction();  connection.Execute(@" insert Member(Username, IsActive) values(@Username, @IsActive)", members, transaction: trans);  trans.Commit(); 
like image 68
kenwarner Avatar answered Oct 14 '22 17:10

kenwarner


I stumbled accross this recently and noticed that the TransactionScope is created after the connection is opened (I assume this since Dappers Execute doesn't open the connection, unlike Query). According to the answer Q4 here: https://stackoverflow.com/a/2886326/455904 that will not result in the connection to be handled by the TransactionScope. My workmate did some quick tests, and opening the connection outside the TransactionScope drastically decreased performance.

So changing to the following should work:

// Assuming the connection isn't already open using (var scope = new TransactionScope()) {     connection.Open();     connection.Execute(@" insert Member(Username, IsActive) values(@Username, @IsActive)", members);      scope.Complete(); } 
like image 35
Fredrik Ljung Avatar answered Oct 14 '22 17:10

Fredrik Ljung