Why does Format File cause error during SQL Server Bulk Insert?

Question

I am trying to bulk insert a CSV file into SQL Server 2017 using an XML Format File:

<?xml version="1.0"?>
<BCPFORMAT xmlns="https://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <RECORD>
        <FIELD ID="Col_0" xsi:type="CharTerm" MAX_LENGTH="100" TERMINATOR=","/>
        <FIELD ID="Col_1" xsi:type="CharTerm" MAX_LENGTH="100" TERMINATOR=","/>
        <FIELD ID="Col_2" xsi:type="CharTerm" MAX_LENGTH="3" TERMINATOR=","/>
        <FIELD ID="Col_3" xsi:type="CharFixed" Length="19" TERMINATOR=","/>
        <FIELD ID="Col_4" xsi:type="CharFixed" Length="19" TERMINATOR=","/>
        <FIELD ID="Col_5" xsi:type="CharFixed" Length="53" TERMINATOR=","/>
        <FIELD ID="Col_6" xsi:type="CharFixed" Length="10" TERMINATOR=","/>
    </RECORD>
    <ROW>
        <COLUMN SOURCE="Col_0" NAME="NAME" xsi:type="SQLVARCHAR"/>
        <COLUMN SOURCE="Col_1" NAME="MATCHNAME" xsi:type="SQLVARCHAR"/>
        <COLUMN SOURCE="Col_2" NAME="CHROMOSOME" xsi:type="SQLVARCHAR"/>
        <COLUMN SOURCE="Col_3" NAME="START_LOCATION" xsi:type="SQLBIGINT"/>
        <COLUMN SOURCE="Col_4" NAME="END_LOCATION" xsi:type="SQLBIGINT"/>
        <COLUMN SOURCE="Col_5" NAME="CENTIMORGANS" xsi:type="SQLFLT4"/>
        <COLUMN SOURCE="Col_6" NAME="MATCHING_SNPS" xsi:type="SQLSMALLINT"/>
    </ROW>
</BCPFORMAT>

When I execute this, I get this error message:

Error during Execute 37000(4855)[Microsoft][ODBC SQL Server Driver][SQL Server]Line 2 in format file "c:\temp\FormatFile.xml": unexpected element "BCPFORMAT".

I'm at a loss to understand why the BCPFORMAT element is not recognized. The XML is properly formatted according to Altova XMYSpy.

The code to perform the upload is:

bulk insert FF_ChromBrwResults_20190124 from 'c:\temp\temp.csv'
 with (FIELDTERMINATOR = ',',FIRSTROW =1,ROWTERMINATOR = '\n'
,KEEPIDENTITY,CODEPAGE ='OEM',KEEPNULLS, FORMATFILE = 'c:\temp\FormatFile.xml' ) 

The CSV has many rows in this format, ending with a linefeed:

Name1,Name2, 1,48745397,54260005,2.84,1000

I've tried executing this from .NET code and in AQT where the ODBC connection is working; both methods produced the same error.

Answer 1

This error occurs because the url for xmlns contains HTTPS correct is HTTP.

INCORRECT = <BCPFORMAT xmlns="https://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

CORRECT = <BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

Your XML file also need one modification transform all CharFixed to CharTerm once the delimiter of CharFixed is not comma but length property.

Change too all type SQLVARCHAR to SQLVARYCHAR

<?xml version="1.0"?>
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <RECORD>
        <FIELD ID="Col_0" xsi:type="CharTerm" MAX_LENGTH="100" TERMINATOR=","/>
        <FIELD ID="Col_1" xsi:type="CharTerm" MAX_LENGTH="100" TERMINATOR=","/>
        <FIELD ID="Col_2" xsi:type="CharTerm" MAX_LENGTH="3" TERMINATOR=","/>
        <FIELD ID="Col_3" xsi:type="CharTerm" TERMINATOR=","/>
        <FIELD ID="Col_4" xsi:type="CharTerm" TERMINATOR=","/>
        <FIELD ID="Col_5" xsi:type="CharTerm" TERMINATOR=","/>
        <FIELD ID="Col_6" xsi:type="CharTerm" TERMINATOR=","/>
    </RECORD>
    <ROW>
        <COLUMN SOURCE="Col_0" NAME="NAME" xsi:type="SQLVARYCHAR"/>
        <COLUMN SOURCE="Col_1" NAME="MATCHNAME" xsi:type="SQLVARYCHAR"/>
        <COLUMN SOURCE="Col_2" NAME="CHROMOSOME" xsi:type="SQLVARYCHAR"/>
        <COLUMN SOURCE="Col_3" NAME="START_LOCATION" xsi:type="SQLBIGINT"/>
        <COLUMN SOURCE="Col_4" NAME="END_LOCATION" xsi:type="SQLBIGINT"/>
        <COLUMN SOURCE="Col_5" NAME="CENTIMORGANS" xsi:type="SQLFLT4"/>
        <COLUMN SOURCE="Col_6" NAME="MATCHING_SNPS" xsi:type="SQLSMALLINT"/>
    </ROW>
</BCPFORMAT> 

Answer 2

Not a complete answer, as I find format files an enduring mystery, but try letting BCP generate the format file, eg:

bcp tempdb.dbo.FF_ChromBrwResults_20190124 format nul -c -x -t "," -f formatfile_gen.xml  -T -S localhost

which for this table

create table FF_ChromBrwResults_20190124
(
  NAME varchar(100),
  MATCHNAME varchar(100),
  CHROMOSOME varchar(30),
  START_LOCATION bigint,
  END_LOCATION bigint,
  CENTIMORGANS real,
  MATCHING_SNPS smallint
)

outputs

<?xml version="1.0"?>
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
 <RECORD>
  <FIELD ID="1" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="100" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
  <FIELD ID="2" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="100" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
  <FIELD ID="3" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="30" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
  <FIELD ID="4" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="21"/>
  <FIELD ID="5" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="21"/>
  <FIELD ID="6" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="30"/>
  <FIELD ID="7" xsi:type="CharTerm" TERMINATOR="\r\n" MAX_LENGTH="7"/>
 </RECORD>
 <ROW>
  <COLUMN SOURCE="1" NAME="NAME" xsi:type="SQLVARYCHAR"/>
  <COLUMN SOURCE="2" NAME="MATCHNAME" xsi:type="SQLVARYCHAR"/>
  <COLUMN SOURCE="3" NAME="CHROMOSOME" xsi:type="SQLVARYCHAR"/>
  <COLUMN SOURCE="4" NAME="START_LOCATION" xsi:type="SQLBIGINT"/>
  <COLUMN SOURCE="5" NAME="END_LOCATION" xsi:type="SQLBIGINT"/>
  <COLUMN SOURCE="6" NAME="CENTIMORGANS" xsi:type="SQLFLT4"/>
  <COLUMN SOURCE="7" NAME="MATCHING_SNPS" xsi:type="SQLSMALLINT"/>
 </ROW>
</BCPFORMAT>