Why does `findstr` with variable expansion in its search string return unexpected results when involved in a pipe?

Question

While trying to provide a comprehensive answer to the question Why is FindStr returning not-found, I encountered strange behaviour of code that involves a pipe. This is some code based on the original question (to be executed in a batch-file):

rem // Set variable `vData` to literally contain `...;%main%\Programs\Go\Bin`:
set "vData=...;%%main%%\Programs\Go\Bin"
set "main=C:\Main"

echo/%vData%| findstr /I /C:"%%main%%\\Programs\\Go\\Bin"

This does not return a match, hence nothing is echoed and ErrorLevel becomes set to 1.

Though when I go through the parsing process step by step, I conclude the opposite, I do expect a match and an ErrorLevel of 0, because:

1. at first, the whole line is parsed and immediate (%) expansion takes place, hence %vData% becomes expanded and %% become replaced by %, resulting in the following command line that is going to be executed:

   echo/...;%main%\Programs\Go\Bin| findstr /I /C:"%main%\\Programs\\Go\\Bin"
   

2. each side of the pipe | is executed in its own new cmd instance by cmd /S /D c, both of which run in cmd context (which affects handling of %-expansion), resulting in these parts:

   • left side of the pipe:

     echo/...;C:\Main\Programs\Go\Bin
     

   • right side of the pipe:

     findstr /I /C:"C:\Main\\Programs\\Go\\Bin"
     

     (the search string that findstr finally uses is C:\Main\Programs\Go\Bin as the \ is used as escape character even in literal search mode, /C or /L)

As you can see the final search string actually occurs within the echoed string, therefore I expect a match, but the command lines does not return one. So what is going on here, what do I miss?

---

When I clear the variable main before executing the pipe command line, I get the expected result, which leads me to the conclusion that the variable main becomes not expanded despite my assumption above (note that in cmd context, %main% is kept literally when the variable is empty). Am I right?

---

It is getting even more confusing: when I put the right side of the pipe in between parentheses, a match is returned, independent on whether or not the variable main is defined:

echo/%vData%| (findstr /I /C:"%%main%%\\Programs\\Go\\Bin")

Can anyone explain this? Could this be connected to the fact that findstr is an external command, opposed to echo? (I mean, break | echo/%vData% expands the value of main as expected if defined...)

Answer 1

I simplified your example to:

@echo off

set "main=abc"
break | findstr /c:"111" %%main%%
break | echo findstr /c:"222" %%main%%

The output is:

FINDSTR: %main% kann nicht geöffnet werden.
findstr /c:"222" abc

This proves that using an exe-file in a pipe results in different behaviour than using an internal batch command.
Only for internal commands a new cmd.exe instance will be created.
That's also the cause why the findstr doesn't expand the percent signs.

This confusing line expands, because the parentheses forces a new cmd.exe instance.

break | (findstr /c:"111" %%main%%)

I will modify the explanation at 5.3 Pipes - How does the Windows Command Interpreter (CMD.EXE) parse scripts?

Answer 2

In a CMD script changes things.

My Quick and Dirty understanding of the Reason:

The batch is behaving as expected, there is no need to double up the percents on main even on the other side of a pipe, because there is no expansion happening, it is a normal variable.

IE the order of operations you write is correct, but also is happening on each line individually, I think you are just so close to the problem you didn't notice or am I not understanding the issue?

I am not sure if this is clearly stated so I plan to review what I believe is happening step by step:

set "vData=...;%%main%%\Programs\Go\Bin"

Result Stored in Memory: "...;%main%\Programs\Go\Bin"

ECHOing %vData% yields: "...;%main%\Programs\Go\Bin"

Add in Main

set "main=C:\Main"

Result Stored in Memory: "C:\Main"

ECHOing %main% yields: "C:\Main"

ECHOing '%%main%%' yields: "%C:\Main%"

Now that Main is defined lets Echo vData Again:

Result Stored in Memory is still: "...;%main%\Programs\Go\Bin"

ECHOing "%vData%" Yields: "...;%main%\Programs\Go\Bin"

CALL ECHOing "%vData%" Yields: "...;C:\Main\Programs\Go\Bin"

So this side was relying on the expansion but the FindStr side isn't because %main% was already defined so it was expanded on the first pass.

When parsed by the CMD interpreter, I believe the sequence will be as follows:

Given your original Find String:

echo/%vData%| findstr /I /C:"%%main%%\\Programs\\Go\\Bin"

Becomes:

echo/...;%main%\Programs\Go\Bin| findstr /I /C:"%C:\Main%\\Programs\\Go\\Bin"

Becomes:

echo/...;C:\Main\Programs\Go\Bin | (findstr /I /C:"%C:\Main%\\Programs\\Go\\Bin")

Alternatively, When parsed by the CMD interpreter, I believe the sequence will be as follows:

Given the modified Regex String:

echo/%vData%| findstr /I /C:"%main%\\Programs\\Go\\Bin"

Becomes:

echo/...;%main%\Programs\Go\Bin| findstr /I /C:"C:\main\\Programs\\Go\\Bin"

Becomes:

echo/...;C:\Main\Programs\Go\Bin | (findstr /I /C:"C:\Main\\Programs\\Go\\Bin")

If your concern is that you were actually storing %%Main%% in a variable, say the %_Regex% Variable:

Then generally you'd have to either Wrap the statements on the other side of the Pipe in a Parenthesis or Call findStr in my experience.

IE:

@(
  SETLOCAL
  ECHO OFF
)
rem // Set variable `vData` to literally contain `...;%main%\Programs\Go\Bin`:
set "vData=...;%%main%%\Programs\Go\Bin"
rem // Set variable `_Regex` to literally contain `%main%\\Programs\\Go\\Bin`:
SET "_Regex=%%main%%\\Programs\\Go\\Bin"
rem // Set variable `_Regex` to literally contain `C:\Main`:
set "main=C:\Main"

ECHO(&ECHO(Variable Contents After Main Set:&ECHO(
     echo(Normal:   vData = "%vData%"
     echo(Normal:  _Regex = "%_Regex%"
     echo(Normal:    main = "%main%"
CALL echo(CALLed:   vData = "%vData%"
CALL echo(CALLed:  _Regex = "%_Regex%"


ECHO(&ECHO(Testing the Results of The FindString Methods:
ECHO(==============================================&ECHO(
ECHO( Original ^%%^%%main^%%^%% Method:
echo/%vData%| findstr /I /C:"%%main%%\\Programs\\Go\\Bin"
ECHO(==============================================&ECHO(
ECHO( Using the ^%%_Regex^%% Stored Variable:
echo/%vData%| findstr /I /C:"%_Regex%"
ECHO(==============================================&ECHO(
ECHO( Using just ^%%main^%%:
echo/%vData%| findstr /I /C:"%main%\\Programs\\Go\\Bin"
ECHO(==============================================&ECHO(
ECHO( Using the CALL ^%%_Regex^%% Stored Variable:
echo/%vData%| CALL findstr /I /C:"%_Regex%"
ECHO(==============================================&ECHO(
ECHO( Using the (^%%_Regex^%%) Stored Variable:
echo/%vData%| ( findstr /I /C:"%_Regex%" )
ECHO(==============================================&ECHO(

Results:

C:\Admin>C:\Admin\TestFindStr.cmd

Variable Contents After Main Set:

Normal:   vData = "...;%main%\Programs\Go\Bin"
Normal:  _Regex = "%main%\\Programs\\Go\\Bin"
Normal:    main = "C:\Main"
CALLed:   vData = "...;C:\Main\Programs\Go\Bin"
CALLed:  _Regex = "C:\Main\\Programs\\Go\\Bin"

Testing the Results of The FindString Methods:
==============================================

 Original %%main%% Method:
==============================================

 Using the %_Regex% Stored Variable:
==============================================

 Using just %main%:
...;C:\Main\Programs\Go\Bin
==============================================

 Using the CALL %_Regex% Stored Variable:
...;C:\Main\Programs\Go\Bin
==============================================

 Using the (%_Regex%) Stored Variable:
...;C:\Main\Programs\Go\Bin
==============================================

This is what I woudl normally expect given my experience with CMD as well.