Windows Batch to read file and parse lines into tokens and variables

Question

I've made a good deal of headway by searching this site and learning the ridiculous language that is Windows batch scripting, but I'm now at a point where I'm stuck. I have a text file with a variable number of lines, each of which looks something like:

AA8315,"United States",N777AN,"American Airlines",AAL98,B772,"Boeing 777-223",AAL,"2013-06-11 23:30:47.923","2013-06-12 00:01:14.459"

My batch file:

set THEDATE=2013-06-12
set THEDATABASE=c:\Kinetic\BaseStation\Basestation.sqb
set THECSVFILE=c:\Flights.csv
set THEOUTPUTFILE=c:\FlightsNew.csv
set THISLINE=""

if exist %THECSVFILE% del %THECSVFILE%
if exist %THEOUTPUTFILE% del %THEOUTPUTFILE%

:: allow time for the csv file to be deleted
timeout /t 2 /nobreak

c:\sqlite3.exe -header -csv %THEDATABASE% "select Aircraft.ModeS, Aircraft.ModeSCountry as Country, Aircraft.Registration as Reg, Aircraft.RegisteredOwners as Owner, Flights.Callsign, Aircraft.ICAOTypeCode as Type, Aircraft.Type as Model, Aircraft.OperatorFlagCode as 'Op Flag', Flights.StartTime as 'First Seen', Flights.EndTime as 'Last Seen' from Aircraft INNER JOIN Flights ON (Aircraft.AircraftID=Flights.AircraftID) where Flights.EndTime like '%THEDATE% %%' order by Flights.EndTime DESC;" >> %THECSVFILE%

::allow time for the csv to be written to file
timeout /t 5 /nobreak

::read %THECSVFILE% and loop through each line
for /F "usebackq tokens=* delims=" %%A in (%THECSVFILE%) do (
    set the_line=%%A
    call :process_line
)

:process_line
for /F "usebackq tokens=1,2,3,4,5,6,7,8,9,10 delims=[,]" %%1 in (%the_line%) do (
    set hexcode=%%1
    set country=%%2
    set reg=%%3
    set owner=%%4
    set callsign=%%5
    set planetype=%%6
    set model=%%7
    set opflag=%%8
    set firstseen=%%9
    set lastseen=%%10
    set THISLINE=%hexcode%,%country%,%reg%,%owner%,%callsign%,%planetype%,%model%,%opflag%,%firstseen%,%lastseen%
    echo %THISLINE% > %THEOUTPUTFILE%
)

(I'm assigning the tokens to variables because I will be doing additional validation and formatting of them later. I need to get this part working first!)

When executed, the script does indeed loop through each line of the file, however it does not seem to be assigning %%1 to the variable hexcode.

The output of the executed command looks like this:

C:\>for /F "usebackq tokens=1,2,3,4,5,6,7,8,9,10 delims=[,]" %1 in (AA8315 "United States" N777AN "American Airlines" AAL98 B772 "Boeing 777-223" AAL "2013-06-11 23:30:47.923" "2013-06-12 00:01:14.459") do (
set hexcode=%1
 set country=%2
 set reg=%3
 set owner=%4
 set callsign=%5
 set planetype=%6
 set model=%7
 set opflag=%8
 set firstseen=%9
 set lastseen=%10
 set THISLINE=,"United States" ,N807FD ,"Fedex Express" ,FDX1378 ,,"Airbus A310-324" ,FDX ,"2013-06-12 22:56:54.639" ,"2013-06-12 23:05:31.822"
 echo ""  1>c:\FlightsNew.csv
)
The system cannot find the file AA8315.

Any help is greatly appreciated!

Answer 1

this works here:

    for /f "tokens=1-10delims=," %%a in ("AA8315,"United States",N777AN,"American Airlines",AAL98,B772,"Boeing 777-223",AAL,"2013-06-11 23:30:47.923","2013-06-12 00:01:14.459"") do (
     set hexcode=%%a
     set country=%%b
     set reg=%%c
     set owner=%%d
     set callsegn=%%e
     set planefype=%%f
     set model=%%g
     set opflag=%%h
     set firstseen=%%i
     set lastseen=%%j
     set THISLINE=%%a,%%b,%%c,%%d,%%e,%%f,%%g,%%h,%%i,%%j
    )
    >"c:\FlightsNew.csv" echo %THISLINE%

I'm not sure, why you need the tokens.