Windows Batch: How to keep empty lines with loop for /f

Question

I'm searching how to keep empty lines when I browse a file with a for loop.

for /f "tokens=1* delims=[" %%i in ('type "test1.txt" ^| find /v /n ""') do (
SET tmp=%%i
echo !tmp! >> test2.txt
)

Actually it works for everybody, but as far as I'm concerned it does not work. For instance if test1.txt content is:

Hello I come from France
I live in Paris

I'm sorry I don't know English, could we speak French please?
If it doesn't bother you

Thank you

Result in test2.txt will be:

[1 
[2 
[3 
[4 
[5 
[6 
[7 

If I put off the "1" near the star "*", result is:

[1]Hello I come from France 
[2]I live in Paris 
[3] 
[4]I'm sorry I don't know English, could we speak French please? 
[5]If it doesn't bother you 
[6] 
[7]Thank you 

Desired output is:

Hello I come from France
I live in Paris

I'm sorry I don't know English, could we speak French please?
If it doesn't bother you

Thank you

Can you please help me to solve this trouble?

Answer 1

This could be done as

@echo off
    setlocal enableextensions disabledelayedexpansion

    for /f "tokens=1,* delims=]" %%a in ('
        find /n /v "" ^< "file1.txt"
    ') do (
        >> "file2.txt" echo(%%b
    )

The output from the inner find command is like

[123]texttexttext

The code uses the closing bracket as delimiter, so the tokens (we are requesting two tokens: 1,* or 1*) are

[123 texttexttext
^    ^
1    2
%%a  %%b

But, as repeated delimiters are handled as only one delimiter, if one line begins with a closing bracket it will be removed. This can be prevented as

@echo off
    setlocal enableextensions disabledelayedexpansion

    for /f "tokens=1,* delims=0123456789" %%a in ('
        find /n /v "" ^< "file1.txt"
    ') do (
        set "line=%%b"
        setlocal enabledelayedexpansion
        >>"file2.txt" echo(!line:~1!
        endlocal
    )

Here the numbers are used as delimiters and the line is tokenized as

[   ]texttexttext
^   ^
%%a %%b

Then the value of the second token is stored inside a variable, with delayed expansion disabled to avoid problems with exclamations inside the data (that will be handled/replaced by the parser if delayed expansion is active)

Once the data is inside the variable, delayed expansion is activated (something needed as we want to retrieve the contents from a variable changed inside a block of code) to output the line from the second position (first character in string is 0) to remove the closing bracket. Once done, delayed expansion is disabled again.

edited as the OP has to incorporate it to a larger/complex script, this code should face the most usual problems

@echo off
    rem For this test we will have delayed expansion from the start
    setlocal enableextensions enabledelayedexpansion

    rem External code block that will make delayed expansion necessary
    if 1==1 ( 
        rem Variables changed inside block
        set "input_file=file1.txt"
        set "output_file=file2.txt"

        rem Grab a reference to the content of the file variables
        for %%i in ("!input_file!") do for %%o in ("!output_file!") do (

            rem Prepare the environment for file work
            setlocal disabledelayedexpansion

            rem Prepare output file
            type nul > "%%~fo"

            rem Process input file and write to output file
            for /f "tokens=1,* delims=0123456789" %%a in ('
                find /n /v "" ^< "%%~fi"
            ') do (
                set "line=%%b"
                setlocal enabledelayedexpansion
                >>"%%~fo" echo(!line:~1!
                endlocal
            )

            rem Restore the previous environment
            endlocal
        )
    )

Answer 2

Here is a slightly different variant using the findstr command rather than find and doing the redirection to the output file file2.txt only once rather than per for /F loop iteration:

@echo off
setlocal EnableExtensions DisableDelayedExpansion
>> "file2.txt" (
    for /F "delims=" %%a in ('findstr /N "^" "file1.txt"') do (
        set "line=%%a"
        setlocal EnableDelayedExpansion
        echo(!line:*:=!
        endlocal
    )
)
endlocal

The findstr command precedes every line by a line number and a colon, like this:

1:Hello I come from France

The sub-string substitution portion !line:*:=! replaces everything up to the first colon (due to *) by nothing, thus removing this.

Replace the >> operator by > in case you want to overwrite an already existing file rather than to append to it.