How to Batch INSERT SQL Server?

Question

I am trying to batch inserting rows from one table to another.

DECLARE @batch INT = 10000;

WHILE @batch > 0
BEGIN
BEGIN TRANSACTION

    INSERT into table2
    select top (@batch) *
    FROM table1

    SET @batch = @@ROWCOUNT

COMMIT TRANSACTION

END

It runs on the first 10,000 and inserts them. Then i get error message "Cannot insert duplicate key" which its trying to insert the same primary key so i assume its trying to repeat the same batch. What logic am i missing here to loop through the batches? Probably something simple but i cant figure it out.

Can anyone help? thanks

Answer 1

Your code keeps inserting the same rows. You can avoid it by "paginating" your inserts:

DECLARE @batch INT = 10000;
DECLARE @page INT = 0
DECLARE @lastCount INT = 1

WHILE @lastCount > 0
BEGIN
BEGIN TRANSACTION

    INSERT into table2
    SELECT col1, col2, ... -- list columns explicitly
    FROM    ( SELECT    ROW_NUMBER() OVER ( ORDER BY YourPrimaryKey ) AS RowNum, *
              FROM      table1
            ) AS RowConstrainedResult
    WHERE   RowNum >= (@page * @batch) AND RowNum < ((@page+1) * @batch)
    SET @lastCount = @@ROWCOUNT
    SET @page = @page + 1

COMMIT TRANSACTION

END

Answer 2

You need some way to eliminate existing rows. You seem to have a primary key, so:

INSERT into table2
   SELECT TOP (@batch) *
   FROM table1 t1
   WHERE NOT EXISTS (SELECT 1 FROM table2 t2 WHERE t2.id = t1.id);