The curious case of the missing exclamation mark in CMD files

Question

I have whittled down a more complex CMD script to the essentials. It reads an input file line by line, unquotes it (if quoted) and writes it out to another CMD file.

The problem is that if the input file contains exclamation marks (! or bang) the character gets stripped out somewhere along the line.

Here is the CMD script, BANG1.CMD:

@echo off
setlocal ENABLEEXTENSIONS ENABLEDELAYEDEXPANSION
if exist bang2.cmd del bang2.cmd
for /f "tokens=*" %%a in (bang1.txt) do call :doit1 %%a
exit /b

:doit1
set P1=%1
if %P1%. EQU . exit /b
call :unquotex P1 %P1%
echo>>bang2.cmd echo P1:[%P1%]
exit /b

:unquotex
set X=%2
set Q=%X:~0,1%
if "!Q!" EQU ^""" SET X=!X:~1,-1!
set %1=%X%
exit /b

Here is the input file BANG1.TXT:

HelloWorld
"Hello World"
Hello!World
"Hello!World"

The resulting file BANG2.CMD ends up containing this:

echo P1:[HelloWorld]
echo P1:[Hello World]
echo P1:[HelloWorld]
echo P1:[HelloWorld]

The question is, what happened to the embedded bangs? I have tried with and without ENABLEDELAYEDEXPANSION. I have even tried escaping (^) the bangs in the input file, still with no luck.

Is there any way to preserve them?

Thanks.

Answer 1

The problem at all is delayed expansion here.
With delayed expansion, exclamation marks are used to expand variables, but when there is only one exclamation mark in a line it will be removed.

Specially in FOR /F loops delayed expansion is tricky to handle, as the expansion of the FOR parameter is directly affected by the delayed expansion. The only solution is to disable it temporarily.
The next problem is the CALL, you can't transfer content with CALL (without destroying it).
It's better to transfer the variable by reference (only the variable name) and then get the content in the called function.
The last problem in your code are the percent expansions, do not use them

when delayed expansion is enabled, as the delayed expansion is evaluated after the percent expansion an expanded line will be expanded a second time by the delayed expansion.
Sample. Assume the content of var is Bang!

echo %var% expands to Bang! but then the delayed expansion will evaluate Bang! to Bang.

With echo !var! you simply get Bang!

@echo off
setlocal ENABLEEXTENSIONS ENABLEDELAYEDEXPANSION
if exist bang2.cmd del bang2.cmd
for /f "tokens=*" %%a in (bang1.txt) do (  
  setlocal DisableDelayedExpansion
  set "line=%%a"
  setlocal EnableDelayedExpansion
  call :doit1 line
  endlocal
  endlocal
)
exit /b

:doit1
set "P1=!%1!"
if "!P1!" EQU "" exit /b
call :unquotex P1
echo>>bang2.cmd echo P1:[!P1!]
exit /b

:unquotex
set "param=!%~1!"
if "!param:~0,1!" == ^""" (
    set "param=!param:~1,-1!"
)
set "%1=!param!"
exit /b

Answer 2

Like this :

@echo off

(for /f "delims=" %%a in ('type bang1.txt') do echo  echo P1:[%%~a])>bang2.cmd