Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How can I load a large flat file into a database table using SSIS?

Tags:

People also ask

How do I load a large file in SSIS?

1. Using Fast Parse for columns in the flat file wherever I could. 2. Using Parallel fast load in SSIS by splitting the output in 4 different streams which load the temp table.

How do I import a flat file into SSIS?

1. Create a new SSIS package and a new Data Flow Task within the package. 2. Drag the Flat File Source adapter from the Toolbox onto the Data Flow workspace and then double-click the Flat File Source to open the Flat File Source Editor.


I'm not sure how it works so I'm looking for the right solution. I think SSIS is the right way to go but I have never used it before

Scenario:

Every morning, I get a tab delimited file with 800K records. I need to load it into my database:

  1. Get file from ftp or local
  2. First, I need to delete the one which not exists in new file from database;
    • How can I compare data in tsql
    • Where should I load data from tab delimited file in order to compare it with the file? Should I use a temp table? ItemID is the unique column in the table.
  3. Second, I need to insert only the new records into the database.
  4. Of course, it should be automated.
  5. It should be efficient way without overheating SQL Database

Don't forget that the file contains 800K records.

Sample flat file data:

ID  ItemID  ItemName  ItemType
--  ------  --------  --------
 1  2345    Apple     Fruit
 2  4578    Banana    Fruit

How can I approach this problem?