Why does delayed expansion fail when inside a piped block of code?

Question

Here is a simple batch file that demonstrates how delayed expansion fails if it is within a block that is being piped. (The failure is toward the end of the script) Can anyone explain why this is?

I have a work-around, but it requires creation of a temporary file. I initially ran into this problem while working on Find files and sort by size in a Windows batch file

@echo off setlocal enableDelayedExpansion  set test1=x set test2=y set test3=z  echo(  echo NORMAL EXPANSION TEST echo Unsorted works (   echo %test3%   echo %test1%   echo %test2% ) echo( echo Sorted works (   echo %test3%   echo %test1%   echo %test2% ) | sort  echo( echo --------- echo(  echo DELAYED EXPANSION TEST echo Unsorted works (   echo !test3!   echo !test1!   echo !test2! ) echo( echo Sorted fails (   echo !test3!   echo !test1!   echo !test2! ) | sort echo( echo Sort workaround (   echo !test3!   echo !test1!   echo !test2! )>temp.txt sort temp.txt del temp.txt 

Here are the results

NORMAL EXPANSION TEST Unsorted works z x y  Sorted works x y z  ---------  DELAYED EXPANSION TEST Unsorted works z x y  Sorted fails !test1! !test2! !test3!  Sort workaround x y z 

Answer 1

As Aacini shows, it seems that many things fail within a pipe.

echo hello | set /p var= echo here | call :function 

But in reality it's only a problem to understand how the pipe works.

Each side of a pipe starts its own cmd.exe in its own ascynchronous thread.
That is the cause why so many things seem to be broken.

But with this knowledge you can avoid this and create new effects

echo one | ( set /p varX= & set varX ) set var1=var2 set var2=content of two echo one | ( echo %%%var1%%% ) echo three | echo MYCMDLINE %%cmdcmdline%% echo four  | (cmd /v:on /c  echo 4: !var2!) 

Update 2019-08-15:
As discovered at Why does `findstr` with variable expansion in its search string return unexpected results when involved in a pipe?, cmd.exe is only used if the command is internal to cmd.exe, if the command is a batch file, or if the command is enclosed in a parenthesized block. External commands not enclosed within parentheses are launched in a new process without the aid of cmd.exe.

EDIT: In depth analysis

As dbenham shows, both sides of the pipes are equivalent for the expansion phases.
The main rules seems to be:

The normal batch parser phases are done
.. percent expansion
.. special character phase/block begin detection
.. delayed expansion (but only if delayed expansion is enabled AND it isn't a command block)

Start the cmd.exe with C:\Windows\system32\cmd.exe /S /D /c"<BATCH COMMAND>"
These expansions follows the rules of the cmd-line parser not the the batch-line parser.

.. percent expansion
.. delayed expansion (but only if delayed expansion is enabled)

The <BATCH COMMAND> will be modified if it's inside a parenthesis block.

( echo one %%cmdcmdline%% echo two ) | more 

Called as C:\Windows\system32\cmd.exe /S /D /c" ( echo one %cmdcmdline% & echo two )", all newlines are changed to & operator.

Why the delayed expansion phase is affected by parenthesis?
I suppose, it can't expand in the batch-parser-phase, as a block can consist of many commands and the delayed expansion take effect when a line is executed.

( set var=one echo !var! set var=two ) | more 

Obviously the !var! can't be evaluated in the batch context, as the lines are executed only in the cmd-line context.

But why it can be evaluated in this case in the batch context?

echo !var! | more 

In my opionion this is a "bug" or inconsitent behaviour, but it's not the first one

EDIT: Adding the LF trick

As dbenham shows, there seems to be some limitation through the cmd-behaviour that changes all line feeds into &.

(   echo 7: part1   rem This kills the entire block because the closing ) is remarked!   echo part2 ) | more 

This results into
C:\Windows\system32\cmd.exe /S /D /c" ( echo 7: part1 & rem This ...& echo part2 ) "
The rem will remark the complete line tail, so even the closing bracket is missing then.

But you can solve this with embedding your own line feeds!

set LF=^   REM The two empty lines above are required (   echo 8: part1   rem This works as it splits the commands %%LF%% echo part2   ) | more 

This results to C:\Windows\system32\cmd.exe /S /D /c" ( echo 8: part1 %cmdcmdline% & rem This works as it splits the commands %LF% echo part2 )"

And as the %lf% is expanded while parsing the parenthises by the parser, the resulting code looks like

( echo 8: part1 & rem This works as it splits the commands    echo part2  ) 

This %LF% behaviour works always inside of parenthesis, also in a batch file.
But not on "normal" lines, there a single <linefeed> will stop the parsing for this line.

EDIT: Asynchronously is not the full truth

I said that the both threads are asynchronous, normally this is true.
But in reality the left thread can lock itself when the piped data isn't consumed by the right thread.
There seems to be a limit of ~1000 characters in the "pipe" buffer, then the thread is blocked until the data is consumed.

@echo off (     (     for /L %%a in ( 1,1,60 ) DO (             echo A long text can lock this thread             echo Thread1 ##### %%a > con         )     )     echo Thread1 ##### end > con ) | (     for /L %%n in ( 1,1,6) DO @(         ping -n 2 localhost > nul         echo Thread2 ..... %%n         set /p x=     ) )