Fastest way to update (populate) 1,000,000 records into a database using .NET

Question

I am using this code to insert 1 million records into an empty table in the database. Ok so without much code I will start from the point I have already interacted with data, and read the schema into a DataTable:

So:

DataTable returnedDtViaLocalDbV11 = DtSqlLocalDb.GetDtViaConName(strConnName, queryStr, strReturnedDtName);

And now that we have returnedDtViaLocalDbV11 lets create a new DataTable to be a clone of the source database table:

DataTable NewDtForBlkInsert = returnedDtViaLocalDbV11.Clone();

Stopwatch SwSqlMdfLocalDb11 = Stopwatch.StartNew();
NewDtForBlkInsert.BeginLoadData();

for (int i = 0; i < 1000000; i++)
{
   NewDtForBlkInsert.LoadDataRow(new object[] { null, "NewShipperCompanyName"+i.ToString(), "NewShipperPhone" }, false);
}
NewDtForBlkInsert.EndLoadData();

DBRCL_SET.UpdateDBWithNewDtUsingSQLBulkCopy(NewDtForBlkInsert, tblClients._TblName, strConnName);

SwSqlMdfLocalDb11.Stop();

var ResSqlMdfLocalDbv11_0 = SwSqlMdfLocalDb11.ElapsedMilliseconds;

This code is populating 1 million records to an embedded SQL database (localDb) in 5200ms. The rest of the code is just implementing the bulkCopy but I will post it anyway.

 public string UpdateDBWithNewDtUsingSQLBulkCopy(DataTable TheLocalDtToPush, string TheOnlineSQLTableName, string WebConfigConName)
 {
    //Open a connection to the database. 
    using (SqlConnection connection = new SqlConnection(ConfigurationManager.ConnectionStrings[WebConfigConName].ConnectionString))
    {
       connection.Open();

       // Perform an initial count on the destination table.
       SqlCommand commandRowCount = new SqlCommand("SELECT COUNT(*) FROM "+TheOnlineSQLTableName +";", connection);
       long countStart = System.Convert.ToInt32(commandRowCount.ExecuteScalar());

       var nl = "\r\n";
       string retStrReport = "";
       retStrReport = string.Concat(string.Format("Starting row count = {0}", countStart), nl);
       retStrReport += string.Concat("==================================================", nl);
       // Create a table with some rows. 
       //DataTable newCustomers = TheLocalDtToPush;

       // Create the SqlBulkCopy object.  
       // Note that the column positions in the source DataTable  
       // match the column positions in the destination table so  
       // there is no need to map columns.  
       using (SqlBulkCopy bulkCopy = new SqlBulkCopy(connection))
       {
          bulkCopy.DestinationTableName = TheOnlineSQLTableName;

          try
          {
             // Write from the source to the destination.
             for (int colIndex = 0; colIndex < TheLocalDtToPush.Columns.Count; colIndex++)
             {
                bulkCopy.ColumnMappings.Add(colIndex, colIndex);
             }
             bulkCopy.WriteToServer(TheLocalDtToPush);
          }

          catch (Exception ex)
          {
             Console.WriteLine(ex.Message);
          }
       }

       // Perform a final count on the destination  
       // table to see how many rows were added. 
       long countEnd = System.Convert.ToInt32(
       commandRowCount.ExecuteScalar());

       retStrReport += string.Concat("Ending row count = ", countEnd, nl);
       retStrReport += string.Concat("==================================================", nl);
       retStrReport += string.Concat((countEnd - countStart)," rows were added.", nl);
       retStrReport += string.Concat("New Customers Was updated successfully", nl, "END OF PROCESS !");
       //Console.ReadLine();
       return retStrReport;
   }
}

Trying it via a connection to SQL server was around 7000ms(at best) & ~7700ms average. Also via a random kv nosql database took around 40 sec (really I did not even keep records of it as it passed over the x2 of sql variants). So... is there a faster way than what I was testing in my code?

Edit

i am using win7 x64 8gb ram and most important i should think (as i5 3ghz) is not so great by now the x3 500Gb Wd on Raid-0 does the job even better but i am just saying if you will check on your pc though just compare it to any other method in your configuration

Answer 1

Have you tried SSIS? I have never written an SSIS package with a loacldb connection, but this is the sort of activity SSIS should be well suited.

If your data source is a SQL Server, another idea would be setting up a linked server. Not sure if this would work with localdb. If you can set up a linked server, you could bypass the C# all together and load your data with an INSERT .. SELECT ... FROM ... SQL statement.

Answer 2

you can use Dapper.NET. Dapper is a micro-ORM, executes a query and map the results to a strongly typed List. Object-relational mapping (ORM, O/RM, and O/R mapping) in computer software is a programming technique for converting data between incompatible type systems in object-oriented programming languages. This creates, in effect, a “virtual object database” that can be used from within the programming language

For more info:

check out https://code.google.com/p/dapper-dot-net/

GitHub Repository: https://github.com/SamSaffron/dapper-dot-net Hope It helps..