Strange behavior with special characters in arguments of batch functions?

Question

Suppose you run test.bat "blabla,blabla,^>blabla", "blaby"

test.bat implementation:

@SETLOCAL
@ECHO OFF
SET list=%~1
ECHO "LIST: %list%"
ECHO "ARG 1: %~1"
ECHO "ARG 2: %~2"
@ENDLOCAL   
@GOTO :EOF

Output is as expected:

"LIST: blabla,blabla,>blabla"
"ARG 1: blabla,blabla,^>blabla"
"ARG 2: blaby"

But what if you make test.bat a function inside a batch file:

@SETLOCAL
CALL :TEST "blabla,blabla,^>blabla", "blaby"
@ENDLOCAL
@GOTO :EOF

:TEST
@SETLOCAL
@ECHO OFF
SET list=%~1
ECHO "LIST: %list%"
ECHO "ARG 1: %~1"
ECHO "ARG 2: %~2"
@ENDLOCAL   
@GOTO :EOF

After running it the output is:

"LIST: blabla,blabla,^"
"ARG 1: blabla,blabla,^^>blabla"
"ARG 2: blaby"

Huh?

1. Where did blabla go in the LIST?
2. ARG 1 has ^^? Why?

Can someone explain how special characters behave differently in function arguments as opposed to command line arguments?

Answer 1

You can get the same result with your 1st batch script simply by using:

call test.bat "blabla,blabla,^>blabla", "blaby"

Your problems stem from an unfortunate aspect of how batch processing parses CALL statements. It is described in phase 6 at How does the Windows Command Interpreter (CMD.EXE) parse scripts?.

Ouch - I thought I understood the caret doubling before, but obviously not. I've heavily edited the following discussion in response to jeb's comments.

The designers of CMD.EXE want a statement like call echo ^^ to give the same result as echo ^^. Both statements reduce ^^ to ^ in phase 2 where special characters are handled. But the CALL statement has to go through phase 1 and phase 2 a second time. So behind the scenes, when CMD.EXE recognizes the CALL statement in phase 6, it doubles the remaining caret back to ^^ and then the second round of phase 2 reduces it back to ^. Both statements echo a single caret to the screen.

Unfortunately CMD.EXE blindly doubles all carets, even if they are quoted. But a quoted caret is not treated as an escape, it is a literal. The carets are no longer consumed. Very unfortunate.

Running call test.bat "blabla,blabla,^>blabla", "blaby" becomes call test.bat "blabla,blabla,^^>blabla" "blaby" in phase 6 of the parser.

That easily explains why ARG 1 looks like it does in your output.

As far as where did blabla go?, that is just a bit trickier.

When your script executes SET list=%~1, the quotes are removed, the ^^ is treated as an escaped caret which reduces to ^, and the > is no longer escaped. So the output of your SET statement is redirected to a "blabla" file. Of course SET has no output so you should have a zero length "blabla" file on your hard drive.

EDIT - How to correctly pass the desired arguments using "late expansion"

In his answer, davor attempted to reverse the effect of the caret doubling within the called routine. But that is not reliable because you cannot know for sure how many times the carets may have been doubled. It is better if you let the caller adjust the call to compensate instead. It is tricky - you have to use what jeb called "late expansion"

Within a batch script you can define a variable that contains the desired argument string, and then delay the expansion until after carets are doubled by escaping the % with another %. You need to double the percents for each CALL in the statement.

@echo off
setlocal
set arg1="blabla,blabla,^>blabla"
call :TEST %%arg1%% "blaby"
echo(
call call :TEST %%%%arg1%%%% "blaby"
::unquoted test
exit /b

:TEST
setlocal
set list=%~1
echo "LIST: %list%"
echo "ARG 1: %~1"
echo "ARG 2: %~2"
exit /b

The above produces the desired result:

"LIST: blabla,blabla,>blabla"
"ARG 1: blabla,blabla,^>blabla"
"ARG 2: blaby"

"LIST: blabla,blabla,>blabla"
"ARG 1: blabla,blabla,^>blabla"
"ARG 2: blaby"

The expansion rules are different when run from the command line. It is impossible to escape a % from the command line. Instead you must add a caret within the percents that prevents the expansion phase from recognizing the name on the 1st pass, and then when the caret is stripped in phase 2, the 2nd pass of expansion properly expands the variable.

The following uses davor's original TEST.BAT

C:\test>test.bat "blabla,blabla,^>blabla" "blaby"
"LIST: blabla,blabla,>blabla"
"ARG 1: blabla,blabla,^>blabla"
"ARG 2: blaby"

C:\test>set arg1="blabla,blabla,^>blabla"

C:\test>test.bat %arg1% "blaby"
"LIST: blabla,blabla,>blabla"
"ARG 1: blabla,blabla,^>blabla"
"ARG 2: blaby"

C:\test>call test.bat %^arg1% "blaby"
"LIST: blabla,blabla,>blabla"
"ARG 1: blabla,blabla,^>blabla"
"ARG 2: blaby"

C:\test>set arg2=%^arg1%

C:\test>call call test.bat %^arg2% "blaby"
"LIST: blabla,blabla,>blabla"
"ARG 1: blabla,blabla,^>blabla"
"ARG 2: blaby"

An alternative to escaping - pass values by reference!

All in all, the escape rules are ridiculously complicated. That is why advanced batch scripting often passes string values by reference instead of as literals. The desired string is placed in a variable and then the name of the variable is passed as an argument. Delayed expansion is used to get the exact string without fear of corruption due to special characters or CALL caret doubling or percent stripping.

Here is a simple test.bat that demonstrates the concept

@echo off
setlocal enableDelayedExpansion
set "var1=!%~1!"
echo var1=!var1!

call :test var1
exit /b

:test
set "var2=!%~1!"
echo var2=!var2!

And here is a demonstration of how it works.

C:\test>set complicatedString="This & that ^" ^& the other thing ^^ is 100% difficult to escape

C:\test>set complicatedString
complicatedString="This & that ^" & the other thing ^ is 100% difficult to escape

C:\test>test.bat complicatedString
var1="This & that ^" & the other thing ^ is 100% difficult to escape
var2="This & that ^" & the other thing ^ is 100% difficult to escape

C:\test>call test.bat complicatedString
var1="This & that ^" & the other thing ^ is 100% difficult to escape
var2="This & that ^" & the other thing ^ is 100% difficult to escape

C:\test>call call test.bat complicatedString
var1="This & that ^" & the other thing ^ is 100% difficult to escape
var2="This & that ^" & the other thing ^ is 100% difficult to escape