Bulk insert fmt text qualifier

Question

I've a BULK INSERT task that takes data from a csv and imports into a table. Only problem is that one of the columns can contain a comma, so the import doesn't work as expected.

I've tried to fix this by creating a format (fmt) file, the contents of which I've detailed below:-

9.0
6
1       SQLCHAR       0       50      ","       1     "Identifier"                                 Latin1_General_CI_AS
2       SQLCHAR       0       50      ","       2     "Name"                                             Latin1_General_CI_AS
3       SQLCHAR       0       50      ","       3     "Date of Birth"                                            Latin1_General_CI_AS
4       SQLCHAR       0       50      ","       4     "Admission"                                                Latin1_General_CI_AS
5       SQLCHAR       0       50      ","       5     "Code"                               Latin1_General_CI_AS
6       SQLCHAR       0       50      "\r\n"   6     "Length"                                                      Latin1_General_CI_AS

The column causing me pain is column 2 "Name".

I've tried a couple of things to identify the column as being text qualified and containing a comma but I'm not getting the result I want.

If I change to the following:-

"\"," - I get something like this -- "Richardson, Mat

This isn't correct, so I tried this instead, as suggested on some other forums / sites:-

"\",\""

This doesn't work at all and actually gives me the error

Cannot obtain the required interface ("IID_IColumnsInfo") from OLE DB provider "BULK" for linked server "(null)".Bulk load: An unexpected end of file was encountered in the data file.

I've tried a few other combinations and just can't get this right. Any help or guidance would be massively appreciated.

Answer 1

Not really answering your question regarding format files but a possible get you working solution.

Format files are incomprehensible arcana from the 1980s to me, bulk insert is uber fussy and unforgiving. Therefore I tend to clean data with a few lines of powershell instead. Here's an example I used recently to convert a CSV to Pipe separated, to remove some random quoting on the output and to allow for commas in the records:

Import-Csv -Path  $dirtyCsv | 
ConvertTo-CSV  -NoType -Delimiter '|' | 
%{ $_.Replace('"','') } | 
Out-File $cleanCsv

You get the idea...

This then simply imported:

BULK INSERT SomeTable FROM 'clean.csv' WITH ( FIRSTROW = 2, FIELDTERMINATOR = '|', ROWTERMINATOR = '\n' )

Hope this helps.

Answer 2

This is occurring because you are telling the bulk insert that your field terminator for the column before name is a simple comma and that the field terminator for the Name column itself is double quote, then comma. You need to change the field terminator for the column before Name to be comma then double quote if you want to take care of the remaining double quote.

I believe your field terminator for the column before name should be: ",\"", where:

• ,=comma

• /" = double quotes

Enclosed in another set of double quotes; it is the value to be used as the field terminator.

Flip the comma and the double quotes for the field terminator of your Name column.

So it should look like this:

9.0
6
1       SQLCHAR       0       50      ",\""       1     "Identifier"                                 Latin1_General_CI_AS
2       SQLCHAR       0       50      "\","       2     "Name"                                             Latin1_General_CI_AS
3       SQLCHAR       0       50      ","       3     "Date of Birth"                                            Latin1_General_CI_AS
4       SQLCHAR       0       50      ","       4     "Admission"                                                Latin1_General_CI_AS
5       SQLCHAR       0       50      ","       5     "Code"                               Latin1_General_CI_AS
6       SQLCHAR       0       50      "\r\n"   6     "Length"