Syntax error in one of two almost-identical batch scripts: ")" cannot be processed syntactically here

Question

I am trying to set up a Jenkins server for automatic Unity builds.

Therefore I have written two (in my eyes) basically identical batch scripts.

Both scripts are run as build steps by Jenkins via an Execute Windows batch command step using

Command: E:\unityImport.bat

and after that a second Execute Windows batch command step using

Command: E:\unityBuild.bat

They both have the same beginning as I need to gather some file paths and in particular the project's unity version. So in both scripts I use exactly the same way of parsing and string splitting the project version. The only thing that is different between them is that the first one starts Unity and imports a dedicated unitypackage (which hold the method to execute in the next step) into the project while the second one again starts Unity to perform the actual build (unfortunately it didn't work in one single go ... Unity seems to try to execute the method before the unitypackage is imported).

However the second script always fails with a syntax error

")" cannot be processed syntactically here.

---

What I try to achieve is

1. Read out the content of the file %WORKSPACE%\ProjectSettings\ProjectVersion.txt

   SET /p TEST=<%WORKSPACE%\ProjectSettings\ProjectVersion.txt
   

   The content of %TEST% usually looks like e.g.

   m_EditorVersion: 2019.3.4f1
   

   and ECHO. ProjectVersion.txt = %TEST% looks like

   ProjectVersion.txt = m_EditorVersion: 2019.3.4f1
   

2. string split in order to only take the last part containing the version number

   for %%x in (%TEST::= %) do (
       SET "VALUE=%%x"
       SET "UNITY_VERSION=!VALUE:~0,-2!" 
   )
   

   so %UNITY_VERSION% usually contains e.g. 2019.3.4. I don't split more away because there are also Unity version with two digits like e.g. 2018.4.18

3. string split on the . in order to only get the major release number

   for /f "tokens=1,2 delims=." %%a in ("%UNITY_VERSION%") do (
       SET "A=%%a"
       SET "B=%%b"
   )
   SET "UNITY_VERSION=%A%.%B%"
   

   which results in %UNITY_VERSION% being e.g. 2019.3

4. Finally search in all installed Unity versions if the required version is present

   set "UNITY_FOLDER="
   for /f "delims=" %%a in ('dir /b E:\Unity\%UNITY_VERSION%*') do (
       set "UNITY_FOLDER=%%a"
   )
   

   after this we either found a valid Unity installation folder for the given version or not.

---

So here are the scripts.

Import (This works as expected)

@ECHO OFF
CLS
ECHO.

cd %WORKSPACE%

IF NOT EXIST %WORKSPACE%\ProjectSettings\ProjectVersion.txt (
    EXIT 1
)

SETLOCAL ENABLEDELAYEDEXPANSION

SET /p TEST=<%WORKSPACE%\ProjectSettings\ProjectVersion.txt
ECHO. ProjectVersion.txt = %TEST%

for %%x in (%TEST::= %) do (
    SET "VALUE=%%x"
    SET "UNITY_VERSION=!VALUE:~0,-2!" 
)

for /f "tokens=1,2 delims=." %%a in ("%UNITY_VERSION%") do (
    SET "A=%%a"
    SET "B=%%b"
)

SET "UNITY_VERSION=%A%.%B%"
ECHO. Project Unity Version = %UNITY_VERSION%

set "UNITY_FOLDER="
for /f "delims=" %%a in ('dir /b E:\Unity\%UNITY_VERSION%*') do (
    set "UNITY_FOLDER=%%a"
)

IF "%UNITY_FOLDER%"=="" (
    EXIT 1
)

ECHO. Using Unity Version %UNITY_FOLDER%

ECHO. Running:
ECHO. E:\Unity\%UNITY_FOLDER%\Editor\Unity.exe -quit -batchmode -projectPath %WORKSPACE% -logFile - -importPackage E:\UnityBuildPackage\AutoBuilder.unitypackage

E:\Unity\%UNITY_FOLDER%\Editor\Unity.exe -quit -batchmode -projectPath %WORKSPACE% -logFile - -importPackage E:\UnityBuildPackage\AutoBuilder.unitypackage

IF NOT %errorlevel% equ 0 (
    EXIT 1
) 

EXIT 0

Build (This fails with a syntax error I will mark with REM HERE IT BREAKS! ... which is not present in the actual script)

@ECHO OFF
CLS
ECHO.

cd %WORKSPACE%

IF NOT EXIST %WORKSPACE%\ProjectSettings\ProjectVersion.txt (
    EXIT 1
)

SETLOCAL ENABLEDELAYEDEXPANSION

SET /p TEST=<%WORKSPACE%\ProjectSettings\ProjectVersion.txt
ECHO. ProjectVersion.txt = %TEST%

REM HERE IT BREAKS! The before echo is the last I see before getting the syntax error

for %%x in (%TEST::= %) do (
    SET "VALUE=%%x"
    SET "UNITY_VERSION=!VALUE:~0,-2!" 
)

for /f "tokens=1,2 delims=." %%a in ("%UNITY_VERSION%") do (
    SET "A=%%a"
    SET "B=%%b"
)

SET "UNITY_VERSION=%A%.%B%"
ECHO. Project Unity Version = %UNITY_VERSION%

set "UNITY_FOLDER="
for /f "delims=" %%a in ('dir /b E:\Unity\%UNITY_VERSION%*') do (
    set "UNITY_FOLDER=%%a"
)

IF "%UNITY_FOLDER%"=="" (
    EXIT 1
)

ECHO. Using Unity Version %UNITY_FOLDER%

...

I don't think the rest matters since as said What I see in the console it already breaks after e.g.

ProjectVersion.txt = 2019.3.4f1

")" cannot be processed syntactically here.

---

Does anyone see the mistake or is there maybe something with Jenkins that makes the second script fail with a syntax error even though as far as I see they are basically identical?

Answer 1

There are multiple small issues with the code which I explain one after the other below my suggestion for the batch file.

The task to get the UNITY_FOLDER according to UNITY_VERSION as defined in file ProjectVersion.txt can be done more efficient by using the following code:

@echo off
setlocal EnableExtensions DisableDelayedExpansion

if not defined WORKSPACE (
    echo ERROR: Environment variable WORKSPACE is not defined.
    exit /B 1
)

if not exist "%WORKSPACE%\ProjectSettings\ProjectVersion.txt" (
    echo ERROR: File "%WORKSPACE%\ProjectSettings\ProjectVersion.txt" does not exist.
    exit /B 1
)

set "UNITY_FOLDER="
set "UNITY_VERSION="
for /F "usebackq tokens=2-4 delims=. " %%I in ("%WORKSPACE%\ProjectSettings\ProjectVersion.txt") do (
    if not "%%~K" == "" (
        for /F "delims=abcdef" %%L in ("%%~K") do (
            set "UNITY_VERSION=%%~I.%%~J.%%~L"
            for /D %%M in ("E:\Unity\%%~I.%%~J*") do set "UNITY_FOLDER=%%M"
        )
    )
)

if not defined UNITY_VERSION (
    echo ERROR: Failed to determine unity version from "%WORKSPACE%\ProjectSettings\ProjectVersion.txt".
    exit /B 1
)
if not defined UNITY_FOLDER (
    echo ERROR: Failed to find a folder in "E:\Unity" for unity version %UNITY_VERSION%.
    exit /B 1
)

echo Found for unity version %UNITY_VERSION% the folder "%UNITY_FOLDER%".

cd /D "%WORKSPACE%" 2>nul
if errorlevel 1 (
    echo ERROR: Failed to set "%WORKSPACE%" as current folder.
    exit /B
)
rem Other commands to execute.

endlocal

This batch file first sets up the execution environment required for this batch file using command SETLOCAL.

The existence of the environment variable WORKSPACE is verified next by the batch file. This environment variable should be defined by Jenkins outside this batch file. An error message is output on missing definition of this important environment variable.

Then the existence of the text file is checked with printing an error message if not existing and exiting batch file with exit code 1.

The two environment variables UNITY_FOLDER and UNITY_VERSION are deleted if defined by chance outside the batch file.

Next the text file is processed which should contain only one non-empty line with the data of interest. Otherwise it would be necessary to change the code to evaluate also the first substring if being equal m_EditorVersion: before execution of the other commands.

FOR with option /F interprets a set enclosed in " by default as string to process. But in this case the string in " should be interpreted as full qualified file name of the file of which contents should be processed line by line by FOR. For that reason the option usebackq is used to get the wanted file contents processing behavior.

FOR ignores always empty lines on processing the contents of a file. So it would not matter if the text file contains at top one or more empty lines.

FOR splits up a line by default into substrings using normal space and horizontal tab character as string delimiters. If the first space/tab delimited string starts with a semicolon being the default end of line character after removing all leading spaces/tabs, the line would be also ignored by FOR like an empty line. Finally just the first space/tab delimited string would be assigned to the specified loop variable I.

This default line processing behavior is not wanted here because of getting just m_EditorVersion: assigned to the specified loop variable I is not enough. For that reason the option delims=.  is used to get the line split up on dots and spaces. The option tokens=2-4 informs FOR that the second space/dot delimited substring 2019 should be assigned to loop variable I, the third space/dot delimited substring 3 to next loop variable J which is the next character in the ASCII table and the fourth space/dot delimited substring 4f1 to next but one loop variable K.

It is important here to specify delims=.  at end of the options argument string with the space character as last character because of the space character is otherwise interpreted as an options separating character to ignore like the space between usebackq and tokens=2-4 and the space between tokens=2-4 and delims=. . In fact it would be also possible to write the options without spaces like "usebackqtokens=2-4delims=. ", but that makes the argument string with the options difficult to read.

The default end of line definition eol=; can be kept here because of the line with the unity version in ProjectVersion.txt does not have a semicolon after 0 or more spaces/dots and is never ignored for that reason.

FOR runs the commands in the command block on having found in line at least the second space/dot delimited string assigned to loop variable I, i.e. a non-empty string is assigned to specified loop variable I. But the commands should be executed only if all three parts of the unity version were determined by FOR and assigned to the loop variables I, J and K. Therefore a simple string comparison is made to verify that loop variable value reference %%~K does not expand to an empty string as that would mean not having read enough parts of unity version from the file.

I don't know what f1 at end of the editor version means. So one more FOR with option /F is used to split the string 4f1 (no usebackq on string enclosed in ") into substrings using the characters abcdef (lower case hexadecimal characters) as string delimiters and get assigned to specified loop variable L just the first substring. That should never fail and so environment variable UNITY_VERSION is defined with 2019.3.4.

The third FOR is executed inside the second FOR although it could be also outside because of not referencing loop variable L. So the following code could be also used here with the same result.

for /F "usebackq tokens=2-4 delims=. " %%I in ("%WORKSPACE%\ProjectSettings\ProjectVersion.txt") do (
    if not "%%~K" == "" (
        for /F "delims=abcdef" %%L in ("%%~K") do set "UNITY_VERSION=%%~I.%%~J.%%~L"
        for /D %%M in ("E:\Unity\%%~I.%%~J*") do set "UNITY_FOLDER=%%M"
    )
)

FOR with option /D and a set containing * (or ?) results in searching in specified directory E:\Unity for a non-hidden directory of which name starts with 2019.3. Each non-hidden directory in E:\Unity matching the wildcard pattern 2019.3* is assigned one after the other with full qualified name (drive + path + name) first to loop variable M and next to environment variable UNITY_FOLDER. FOR never encloses itself a file/folder string in " which is the reason why %%M can be used here and %%~M is not necessary. The folder name assigned to loop variable M is never enclosed in " in this case. So the environment variable UNITY_FOLDER contains the last folder matching the wildcard pattern returned by the file system with full path. This means on multiple folder names matching the wildcard pattern 2019.3* that the file system determines which folder name is assigned last to UNITY_FOLDER. NTFS stores directory entries in its master file table sorted in a local specific alphabetic order while FAT, FAT32 and exFAT store directory entries unsorted in their file allocation tables.

Note: If the third number of editor version is not really needed as it looks like according to the code in question, it would be also possible to use:

for /F "usebackq tokens=2-4 delims=. " %%I in ("%WORKSPACE%\ProjectSettings\ProjectVersion.txt") do (
    if not "%%~J" == "" (
        set "UNITY_VERSION=%%~I.%%~J"
        for /D %%K in ("E:\Unity\%%~I.%%~J*") do set "UNITY_FOLDER=%%K"
    )
)

Two additional checks are made if the code could successfully determine unity version and find a matching unity folder.

The echo command line at bottom of the batch file is just for verification of the result on running this batch file with WORKSPACE defined outside the batch file in command prompt window and everything worked as expected.

There is no need to make the workspace directory the current directory up to end of the batch file, but I added the code to do that with the verification if changing the current directory to workspace directory was done really successfully.

---

Issue 1: File/folder argument strings not enclosed in quotes

The help output on running in a command prompt cmd /? explains with last paragraph on last page that a file/folder argument string containing a space or one of these characters &()[]{}^=;!'+,`~ requires surrounding straight double quotes. So it is advisable to always enclose file/folder names without or with path in ", especially on one or more parts being dynamically defined by an environment variable or being read from file system.

So not good are the following command lines:

cd %WORKSPACE%
IF NOT EXIST %WORKSPACE%\ProjectSettings\ProjectVersion.txt
SET /p TEST=<%WORKSPACE%\ProjectSettings\ProjectVersion.txt

Better would be using:

cd "%WORKSPACE%"
IF NOT EXIST "%WORKSPACE%\ProjectSettings\ProjectVersion.txt"
SET /p TEST=<"%WORKSPACE%\ProjectSettings\ProjectVersion.txt"

It can be read in short help output on running cd /? that the command CD does not interpret a space character as argument separator like it is the case for most other internal commands of Windows command processor cmd.exe or executables in directory %SystemRoot%\System32 which are installed by default and also belong to the Windows commands according to Microsoft. But changing the current directory fails on omitting " if the directory path contains by chance an ampersand because of & outside a double quoted argument string is interpreted already by cmd.exe as AND operator before execution of CD as described for example in my answer on single line with multiple commands.

It is best to use surrounding " on every argument string which could contain a space or &()[]{}^=;!'+,`~ or the redirection operators <>| which should be interpreted by Windows command processor as literal characters of an argument string. Well, the square brackets do not have anymore a special meaning for Windows command processor. [] are in the list for historical reasons as COMMAND.COM of the first versions of MS-DOS interpreted them not always as literal characters.

---

Issue 2: Usage of a command block for a single command

The Windows command processor is designed primary for

• opening a batch file,
• reading a line from the batch file from previously remembered byte offset or offset 0 on first line,
• parsing and preprocessing this line,
• closing the batch file on no more lines to read,
• remembering current byte offset in batch file,
• executing the command line.

The help output for command IF on running if /? shows at top of first page the general syntax on which the command to execute on condition being true is on the same line as the command IF. The help output for command FOR on running for /? shows at top of first page the general syntax on which the command to execute on each loop iteration is on the same line as the command FOR. Therefore this recommended syntax should be used for an IF condition and a FOR loop on which just one command needs to be executed.

Let us look how Windows command processor interprets the following IF condition with environment variable WORKSPACE being defined with C:\Temp:

IF NOT EXIST %WORKSPACE%\ProjectSettings\ProjectVersion.txt (
    EXIT 1
)

A batch file with just those three lines results in execution of:

IF NOT EXIST C:\Temp\ProjectSettings\ProjectVersion.txt (EXIT 1 )

So Windows command processor detected that there is a command block starting with (, read more lines from the batch file up to matching ), found out that the command block consists of only one command line, and merged the three lines together to one command line for that reason.

So the batch file processing could be speed up a very little bit by writing in the batch file:

IF NOT EXIST "%WORKSPACE%\ProjectSettings\ProjectVersion.txt" EXIT /B 1

Then less CPU instructions are needed to get executed by cmd.exe.

IF NOT EXIST "C:\Temp\ProjectSettings\ProjectVersion.txt" EXIT /B 1

However, the usage of a command block is always possible to make the code of a batch file better readable.

It could be even useful to put entire code of a batch file or a part of it executed often into one command block if that is possible to avoid lots of file open, read, close operations on the batch file which has sometimes a dramatic effect on total execution time as demonstrated by Why is a GOTO loop much slower than a FOR loop and depends additionally on power supply?

See also How does the Windows Command Interpreter (CMD.EXE) parse scripts?

---

Issue 3: ECHO. could result in unwanted behavior

The DosTips forum topic ECHO. FAILS to give text or blank line - Instead use ECHO/ explains that ECHO. could fail to output text or an empty line. The usage of ECHO/ is better if the next character is not ? and best is ECHO(.

The character separating the command ECHO from the string to output can be the standard argument separator space if there is guaranteed that there is a text to output after ECHO  like on ECHO ProjectVersion.txt = %TEST%.

ECHO/ is good to output an empty line.

ECHO( is best if there is next an environment variable reference or a loop variable reference on which was not made sure before that the environment variable is defined at all or the loop variable exists with a non-empty string not starting with a question mark.

---

Issue 4: Usage of SET /P to read a line from a text file

It is possible to use set /P to read the first line from a text file and assign this line to an environment variable as done with:

SET /p TEST=<%WORKSPACE%\ProjectSettings\ProjectVersion.txt

But the text file must have the text to assign to the environment variable at top of the file. An empty line at top of the text file results in assigning nothing to the environment variable which means that if the environment variable TEST is defined already, its value is not changed at all, and if the environment variable TEST is not defined before, it is still not defined after execution of SET.

It is better to use the command FOR with option /F to process the contents of a text file.

---

Issue 5: Usage of command EXIT without option /B

The command EXIT exits the Windows command process which is processing the batch file. It always works, but it should be nevertheless avoided to use EXIT without option /B in most batch files.

A batch file on which EXIT without /B without or with an exit code is executed by cmd.exe results in cmd.exe always terminating itself, even on cmd.exe being started implicit or explicit with option /K to keep the command process running after finishing execution of a command, command line or batch file and independent on the batch file calling hierarchy.

A batch file with EXIT without option /B is therefore hard to debug because of even on running the batch file from within a command prompt window instead of double clicking on it to see error messages, the command process and the console window are closed on cmd.exe reaches the command line with EXIT.

---

Issue 6: Batch file depends on environment defined outside

A good designed batch file does not depend on an execution environment defined outside of the batch file. The two batch files use commands with features available only with enabled command extensions. The command extensions are enabled by default and delayed environment variable expansion is disabled by default, but it is nevertheless better when a batch file defines itself the execution environment and restores the previous execution environment before exiting. This makes sure that the batch file always works as designed even if another batch file calling this batch file sets up a different execution environment.

So after @echo off to make sure that the ECHO mode is turned off the next command line should be:

setlocal EnableExtensions DisableDelayedExpansion

Then the batch file is definitely executed in the expected environment. The command endlocal should be at end of the batch file to restore initial execution environment. But Windows command processor implicitly runs endlocal before exiting the processing of a batch file for every executed setlocal without the execution of a matching endlocal before exiting the batch file processing.

The executions of setlocal /? and endlocal /? result in displaying the help of those two commands. A better explanation can be found in this answer with much more details about the commands SETLOCAL and ENDLOCAL.

The usage of setlocal at top of a batch file to set up the required execution environment and endlocal at bottom of the batch file to restore initial execution environment must be just done wisely in case of a batch file should return results via environment variables to initial execution environment like a parent batch file which called the currently executed batch file.

---

Issue 7: Usage of letters ADFNPSTXZadfnpstxz as loop variable

The help of command FOR output on running for /? describes the modifiers which can be used on referencing the value of a loop variable.

   %~I         - expands %I removing any surrounding quotes (")
   %~fI        - expands %I to a fully qualified path name
   %~dI        - expands %I to a drive letter only
   %~pI        - expands %I to a path only
   %~nI        - expands %I to a file name only
   %~xI        - expands %I to a file extension only
   %~sI        - expanded path contains short names only
   %~aI        - expands %I to file attributes of file
   %~tI        - expands %I to date/time of file
   %~zI        - expands %I to size of file
   %~$PATH:I   - searches the directories listed in the PATH
                  environment variable and expands %I to the
                  fully qualified name of the first one found.
                  If the environment variable name is not
                  defined or the file is not found by the
                  search, then this modifier expands to the
                  empty string

The modifiers can be combined to get compound results:

   %~dpI       - expands %I to a drive letter and path only
   %~nxI       - expands %I to a file name and extension only
   %~fsI       - expands %I to a full path name with short names only
   %~dp$PATH:I - searches the directories listed in the PATH
                  environment variable for %I and expands to the
                  drive letter and path of the first one found.
   %~ftzaI     - expands %I to a DIR like output line

The modifiers are interpreted case-insensitive which means %~FI is the same as %~fI while the loop variable is interpreted always case-sensitive which means loop variable I is interpreted different to loop variable i.

It is advisable to avoid the letters ADFNPSTXZadfnpstxz as loop variable although these letters can be also used as loop variable, especially if a loop variable reference is concatenated with a string as in the batch file command line example below.

for %%x in ("1" 2,3;4) do echo %%~xx5 = ?

The same example for execution directly in a command prompt window:

for %x in ("1" 2,3;4) do @echo %~xx5 = ?

The output is in general (not always):

5 = ?
5 = ?
5 = ?
5 = ?

But the output makes more sense on using I in a batch file:

for %%I in ("1" 2,3;4) do echo %%~Ix5 = ?

The same command line for execution directly in a command prompt window:

for %I in ("1" 2,3;4) do @echo %~Ix5 = ?

The output is in this case always:

1x5 = ?
2x5 = ?
3x5 = ?
4x5 = ?

So it is not possible to use ADFNPSTXZadfnpstxz as loop variable if

1. the loop variable value is referenced with a modifier which means the loop variable value reference begins with %~ (command prompt window) or %%~ (batch file) and
2. the loop variable value reference is concatenated with a string of which first character is identical to the letter used for the loop variable.

So working fine in a command prompt window are:

for %x in (1 2,3;4) do @echo %xx5 = ?      & rem Condition 1 is not true.
for %n in ("1" 2,3;4) do @echo %~nx5 = ?   & rem Condition 2 is not true.
for %x in ("1" 2,3;4) do @echo %~x+5 = ?   & rem Condition 2 is not true.

However, the readability is not good on using a letter which can be used to reference the string value assigned to a loop variable with a modifier.

Readability example for usage in a command prompt window:

for %i in (*) do @echo %~si
for %f in (*) do @echo %~sf
for %i in (*) do @echo %~sni
for %f in (*) do @echo %~snf

In this case i and f work both and the output is the same independent on usage of i or f. But it is easier to see what are the modifiers (s and n) and what is the loop variable on using i and not f for the loop variable.

It is also possible to use other ASCII characters than letters with no special meaning for Windows command processor like # as loop variable if not using FOR with option /F on which multiple substrings are assigned to multiple loop variables.

---

Issue 8: Processing of a set without wildcards by FOR

Let us look on what really happens on using the following code:

setlocal EnableExtensions EnableDelayedExpansion
set "TEST=m_EditorVersion: 2019.3.4f1"
for %%x in (%TEST::= %) do (
    SET "VALUE=%%x"
    SET "UNITY_VERSION=!VALUE:~0,-2!" 
)
endlocal

The string substitution %TEST::= % results in replacing each colon by a space in the string assigned to environment variable TEST on parsing the FOR command line with its command block. So the string

m_EditorVersion: 2019.3.4f1

becomes

m_EditorVersion  2019.3.4f1

Next Windows command processor replaces the two spaces between m_EditorVersion and 2019.3.4f1 by a single space as cleanup. So the set to process by for is finally after parsing and preprocessing the command line with for and its command block:

m_EditorVersion 2019.3.4f1

This set contains neither * nor ?. For that reason the command FOR interprets the set as two simple space separated strings to assign to specified loop variable x one after the other and execute the commands in command block two times for those two strings.

On first iteration m_EditorVersion is assigned to environment variable VALUE and m_EditorVersi to the environment variable UNITY_VERSION. That is not really wanted, but FOR runs the two commands once more, this time with 2019.3.4f1 assigned to the loop variable x. So on second loop iteration 2019.3.4f1 is assigned to the environment variable VALUE and 2019.3.4 to the environment variable UNITY_VERSION.

UNITY_VERSION is defined finally with the wanted string, but that could be done better as shown and explained at top of this answer.

It is not really clear for me why the for command line results in the error message:

")" cannot be processed syntactically here.

That should not happen ever for this FOR loop on m_EditorVersion: 2019.3.4f1 being assigned to environment variable TEST.

Either TEST is defined with a string resulting in the syntax error on execution of the second batch file although that should not be the case according to the description or there is an issue with ( interpreted as beginning of a command block and the Windows command processor fails to find the matching ) which marks the end of the command block.