Faster SQL Inserts?

Question

I'm dealing with chunks of data that are 50k rows each. I'm inserting them into an SQL database using LINQ:

for(int i=0;i<50000;i++)
{
    DB.TableName.InsertOnSubmit
    (
        new TableName
        {
            Value1 = Array[i,0],
            Value2 = Array[i,1]
        }
    );
}
DB.SubmitChanges();

This takes about 6 minutes, and I want it to take much less if possible. Any suggestions?

Answer 1

if you are reading in a file you'd be better off using BULK INSERT (Transact-SQL) and if you are writing that much (50K rows) at one time from memory, you might be better off writing to a flat file first and then using Bulk Insert on that file.

Answer 2

As you are doing a simple insert and not gaining much from the use of LinqToSql, have a look at SqlBulkCopy, it will remove most of the round trips and reduce the overhead on the Sql Server side as well. You will have to make very few coding changes to use it.

Also look at pre-sorting your data by the column that the table is indexed on, as this will lead to better cache hits when SQL-Server is update the table.

Also consider if you should upload the data to a temp staging table that is not indexed, then a stored proc to insert into the main table with a single sql statement. This may let SqlServer spread the indexing work over all your CPUs.