Saving 'tree /f /a" results to a textfile with unicode support

Question

I'm trying to use the tree command in a windows commandline to generate a text file listing the contents of a directory but when I pipe the output the unicode characters get stuffed up.

Here is the command I am using:

tree /f /a > output.txt

The results in the console window are fine:

\---Erika szobája
        cover.jpg
        Erika szobája.m3u
        Kátai Tamás - 01 Télvíz.ogg
        Kátai Tamás - 02 Zölderdõ.ogg
        Kátai Tamás - 03 Renoir kertje.ogg
        Kátai Tamás - 04 Esõben szaladtál.ogg
        Kátai Tamás - 05 Ázik az út.ogg
        Kátai Tamás - 06 Sûrû völgyek takaród.ogg
        Kátai Tamás - 07 Õszhozó.ogg
        Kátai Tamás - 08 Mécsvilág.ogg
        Kátai Tamás - 09 Zúzmara.ogg

But the text file is no good:

\---Erika szob ja
        cover.jpg
        Erika szob ja.m3u
        K tai Tam s - 01 T‚lv¡z.ogg
        K tai Tam s - 02 Z”lderdä.ogg
        K tai Tam s - 03 Renoir kertje.ogg
        K tai Tam s - 04 Esäben szaladt l.ogg
        K tai Tam s - 05 µzik az £t.ogg
        K tai Tam s - 06 S–r– v”lgyek takar¢d.ogg
        K tai Tam s - 07 åszhoz¢.ogg
        K tai Tam s - 08 M‚csvil g.ogg
        K tai Tam s - 09 Z£zmara.ogg

How can I fix this? Ideally the text file would be exactly the same as the output in the console window.

I tried Chris Jester-Young's suggestion (what happened, did you delete it Chris?) of running the command line with the /U switch, it looked like exactly what I needed but it does not appear to work. I have tried opening the file in both VS2008 and notepad and both show the same incorrect characters.

Answer 1

Have someone already tried this:

tree /f /a |clip

Open notepad, ctrl + V, save in notepad as output.txt with unicode support?

Answer 2

Use PowerShell:

powershell -command "tree /f > tree.txt"

Test case:

create.ps1:

mkdir "Erika szobája"
$null | Set-Content "Erika szobája/cover.jpg"
$null | Set-Content "Erika szobája/Erika szobája.m3u"
$null | Set-Content "Erika szobája/Kátai Tamás - 01 Télvíz.ogg"
$null | Set-Content "Erika szobája/Kátai Tamás - 02 Zölderdõ.ogg"
$null | Set-Content "Erika szobája/Kátai Tamás - 03 Renoir kertje.ogg"
$null | Set-Content "Erika szobája/Kátai Tamás - 04 Esõben szaladtál.ogg"
$null | Set-Content "Erika szobája/Kátai Tamás - 05 Ázik az út.ogg"
$null | Set-Content "Erika szobája/Kátai Tamás - 06 Sûrû völgyek takaród.ogg"
$null | Set-Content "Erika szobája/Kátai Tamás - 07 Õszhozó.ogg"
$null | Set-Content "Erika szobája/Kátai Tamás - 08 Mécsvilág.ogg"
$null | Set-Content "Erika szobája/Kátai Tamás - 09 Zúzmara.ogg"

Output:

tree.txt:

Folder PATH listing
Volume serial number is 00000000 0000:0000
C:.
│   create.ps1
│   tree.txt
│   
└───Erika szobája
        cover.jpg
        Erika szobája.m3u
        Kátai Tamás - 01 Télvíz.ogg
        Kátai Tamás - 02 Zölderdo.ogg
        Kátai Tamás - 03 Renoir kertje.ogg
        Kátai Tamás - 04 Esoben szaladtál.ogg
        Kátai Tamás - 05 Azik az út.ogg
        Kátai Tamás - 06 Sûrû völgyek takaród.ogg
        Kátai Tamás - 07 Oszhozó.ogg
        Kátai Tamás - 08 Mécsvilág.ogg
        Kátai Tamás - 09 Zúzmara.ogg

EDIT:

Enhanced and improved version for power users

Test case:

$null | Set-Content "欲速则不达.txt"
$null | Set-Content "爱不是占有,是欣赏.txt"
$null | Set-Content "您先请是礼貌.txt"
$null | Set-Content "萝卜青菜，各有所爱.txt"
$null | Set-Content "广交友，无深交.txt"
$null | Set-Content "一见钟情.txt"
$null | Set-Content "山雨欲来风满楼.txt"

$null | Set-Content "悪妻は百年の不作。.txt"
$null | Set-Content "残り物には福がある。.txt"
$null | Set-Content "虎穴に入らずんば虎子を得ず。.txt"
$null | Set-Content "夏炉冬扇.txt"
$null | Set-Content "花鳥風月.txt"
$null | Set-Content "起死回生.txt"
$null | Set-Content "自業自得.txt"

$null | Set-Content "아는 길도 물어가라.txt"
$null | Set-Content "빈 수레가 요란하다.txt"
$null | Set-Content "방귀뀐 놈이 성낸다.txt"
$null | Set-Content "뜻이 있는 곳에 길이 있다.txt"
$null | Set-Content "콩 심은데 콩나고, 팥 심은데 팥난다.txt"

From his answer, @Chris Jester-Young wrote:

Now, in ulib, the WriteString method is implemented in two classes, SCREEN and STREAM. The SCREEN version uses WriteConsoleW directly, so all the Unicode characters get correctly displayed. The STREAM version converts the Unicode text to one of three different encodings (_UseConsoleConversions ⇒ console codepage (GetConsoleCP), _UseAnsiConversions ⇒ default ANSI codepage, otherwise ⇒ default OEM codepage), and then writes this out.

This means that we cannot rely on getting the characters from a stream. File redirections won't work. We have to rely on writing to the console to get the Unicode characters.

The workaround, or hack, is to write the tree to the console and then dump the buffer to a file.

I have written the scripts to add the tree context menu when you right click on directories in Explorer. Save the files in the same directory and then run Install list menu.bat as administrator to install.

Install list menu.bat

@echo on

regedit /s "List files.reg"

copy "List.ps1" "%SystemRoot%"

pause

List files.reg

Windows Registry Editor Version 5.00

; Directory.
[HKEY_LOCAL_MACHINE\Software\Classes\Directory\Shell\List]
"MUIVerb"="List"
"ExtendedSubCommandsKey"="Directory\\ContextMenus\\List"

[HKEY_LOCAL_MACHINE\Software\Classes\Directory\ContextMenus\List\Shell\Files]
"MUIVerb"="Files"

[HKEY_LOCAL_MACHINE\Software\Classes\Directory\ContextMenus\List\Shell\Files\Command]
; powershell -executionPolicy bypass "%SystemRoot%\List.ps1" -type 'files' -directory '%1'
@=hex(2):70,00,6f,00,77,00,65,00,72,00,73,00,68,00,65,00,6c,00,6c,00,20,00,2d,\
  00,65,00,78,00,65,00,63,00,75,00,74,00,69,00,6f,00,6e,00,50,00,6f,00,6c,00,\
  69,00,63,00,79,00,20,00,62,00,79,00,70,00,61,00,73,00,73,00,20,00,22,00,25,\
  00,53,00,79,00,73,00,74,00,65,00,6d,00,52,00,6f,00,6f,00,74,00,25,00,5c,00,\
  4c,00,69,00,73,00,74,00,2e,00,70,00,73,00,31,00,22,00,20,00,2d,00,74,00,79,\
  00,70,00,65,00,20,00,27,00,66,00,69,00,6c,00,65,00,73,00,27,00,20,00,2d,00,\
  64,00,69,00,72,00,65,00,63,00,74,00,6f,00,72,00,79,00,20,00,27,00,25,00,31,\
  00,27,00,00,00

[HKEY_LOCAL_MACHINE\Software\Classes\Directory\ContextMenus\List\Shell\FilesRecursively]
"MUIVerb"="Files recursively"

[HKEY_LOCAL_MACHINE\Software\Classes\Directory\ContextMenus\List\Shell\FilesRecursively\Command]
; powershell -executionPolicy bypass "%SystemRoot%\List.ps1" -type 'filesRecursively' -directory '%1'
@=hex(2):70,00,6f,00,77,00,65,00,72,00,73,00,68,00,65,00,6c,00,6c,00,20,00,2d,\
  00,65,00,78,00,65,00,63,00,75,00,74,00,69,00,6f,00,6e,00,50,00,6f,00,6c,00,\
  69,00,63,00,79,00,20,00,62,00,79,00,70,00,61,00,73,00,73,00,20,00,22,00,25,\
  00,53,00,79,00,73,00,74,00,65,00,6d,00,52,00,6f,00,6f,00,74,00,25,00,5c,00,\
  4c,00,69,00,73,00,74,00,2e,00,70,00,73,00,31,00,22,00,20,00,2d,00,74,00,79,\
  00,70,00,65,00,20,00,27,00,66,00,69,00,6c,00,65,00,73,00,52,00,65,00,63,00,\
  75,00,72,00,73,00,69,00,76,00,65,00,6c,00,79,00,27,00,20,00,2d,00,64,00,69,\
  00,72,00,65,00,63,00,74,00,6f,00,72,00,79,00,20,00,27,00,25,00,31,00,27,00,\
  00,00

[HKEY_LOCAL_MACHINE\Software\Classes\Directory\ContextMenus\List\Shell\Tree]
"MUIVerb"="Tree"

[HKEY_LOCAL_MACHINE\Software\Classes\Directory\ContextMenus\List\Shell\Tree\Command]
; powershell -executionPolicy bypass "%SystemRoot%\List.ps1" -type 'tree' -directory '%1'
@=hex(2):70,00,6f,00,77,00,65,00,72,00,73,00,68,00,65,00,6c,00,6c,00,20,00,2d,\
  00,65,00,78,00,65,00,63,00,75,00,74,00,69,00,6f,00,6e,00,50,00,6f,00,6c,00,\
  69,00,63,00,79,00,20,00,62,00,79,00,70,00,61,00,73,00,73,00,20,00,22,00,25,\
  00,53,00,79,00,73,00,74,00,65,00,6d,00,52,00,6f,00,6f,00,74,00,25,00,5c,00,\
  4c,00,69,00,73,00,74,00,2e,00,70,00,73,00,31,00,22,00,20,00,2d,00,74,00,79,\
  00,70,00,65,00,20,00,27,00,74,00,72,00,65,00,65,00,27,00,20,00,2d,00,64,00,\
  69,00,72,00,65,00,63,00,74,00,6f,00,72,00,79,00,20,00,27,00,25,00,31,00,27,\
  00,00,00

; Directory background.
[HKEY_LOCAL_MACHINE\Software\Classes\Directory\Background\Shell\List]
"MUIVerb"="List"
"ExtendedSubCommandsKey"="Directory\\Background\\ContextMenus\\List"

[HKEY_LOCAL_MACHINE\Software\Classes\Directory\Background\ContextMenus\List\Shell\Files]
"MUIVerb"="Files"

[HKEY_LOCAL_MACHINE\Software\Classes\Directory\Background\ContextMenus\List\Shell\Files\Command]
; powershell -executionPolicy bypass "%SystemRoot%\List.ps1" -type 'files' -directory '%V'
@=hex(2):70,00,6f,00,77,00,65,00,72,00,73,00,68,00,65,00,6c,00,6c,00,20,00,2d,\
  00,65,00,78,00,65,00,63,00,75,00,74,00,69,00,6f,00,6e,00,50,00,6f,00,6c,00,\
  69,00,63,00,79,00,20,00,62,00,79,00,70,00,61,00,73,00,73,00,20,00,22,00,25,\
  00,53,00,79,00,73,00,74,00,65,00,6d,00,52,00,6f,00,6f,00,74,00,25,00,5c,00,\
  4c,00,69,00,73,00,74,00,2e,00,70,00,73,00,31,00,22,00,20,00,2d,00,74,00,79,\
  00,70,00,65,00,20,00,27,00,66,00,69,00,6c,00,65,00,73,00,27,00,20,00,2d,00,\
  64,00,69,00,72,00,65,00,63,00,74,00,6f,00,72,00,79,00,20,00,27,00,25,00,56,\
  00,27,00,00,00

[HKEY_LOCAL_MACHINE\Software\Classes\Directory\Background\ContextMenus\List\Shell\FilesRecursively]
"MUIVerb"="Files recursively"

[HKEY_LOCAL_MACHINE\Software\Classes\Directory\Background\ContextMenus\List\Shell\FilesRecursively\Command]
; powershell -executionPolicy bypass "%SystemRoot%\List.ps1" -type 'filesRecursively' -directory '%V'
@=hex(2):70,00,6f,00,77,00,65,00,72,00,73,00,68,00,65,00,6c,00,6c,00,20,00,2d,\
  00,65,00,78,00,65,00,63,00,75,00,74,00,69,00,6f,00,6e,00,50,00,6f,00,6c,00,\
  69,00,63,00,79,00,20,00,62,00,79,00,70,00,61,00,73,00,73,00,20,00,22,00,25,\
  00,53,00,79,00,73,00,74,00,65,00,6d,00,52,00,6f,00,6f,00,74,00,25,00,5c,00,\
  4c,00,69,00,73,00,74,00,2e,00,70,00,73,00,31,00,22,00,20,00,2d,00,74,00,79,\
  00,70,00,65,00,20,00,27,00,66,00,69,00,6c,00,65,00,73,00,52,00,65,00,63,00,\
  75,00,72,00,73,00,69,00,76,00,65,00,6c,00,79,00,27,00,20,00,2d,00,64,00,69,\
  00,72,00,65,00,63,00,74,00,6f,00,72,00,79,00,20,00,27,00,25,00,56,00,27,00,\
  00,00

[HKEY_LOCAL_MACHINE\Software\Classes\Directory\Background\ContextMenus\List\Shell\Tree]
"MUIVerb"="Tree"

[HKEY_LOCAL_MACHINE\Software\Classes\Directory\Background\ContextMenus\List\Shell\Tree\Command]
; powershell -executionPolicy bypass "%SystemRoot%\List.ps1" -type 'tree' -directory '%V'
@=hex(2):70,00,6f,00,77,00,65,00,72,00,73,00,68,00,65,00,6c,00,6c,00,20,00,2d,\
  00,65,00,78,00,65,00,63,00,75,00,74,00,69,00,6f,00,6e,00,50,00,6f,00,6c,00,\
  69,00,63,00,79,00,20,00,62,00,79,00,70,00,61,00,73,00,73,00,20,00,22,00,25,\
  00,53,00,79,00,73,00,74,00,65,00,6d,00,52,00,6f,00,6f,00,74,00,25,00,5c,00,\
  4c,00,69,00,73,00,74,00,2e,00,70,00,73,00,31,00,22,00,20,00,2d,00,74,00,79,\
  00,70,00,65,00,20,00,27,00,74,00,72,00,65,00,65,00,27,00,20,00,2d,00,64,00,\
  69,00,72,00,65,00,63,00,74,00,6f,00,72,00,79,00,20,00,27,00,25,00,56,00,27,\
  00,00,00

List.ps1

function sortNaturally {
    [Regex]::replace($_, '\d+', {
        $args[0].value.padLeft(20)
    })
}

function writeList {
    param(
        [parameter(mandatory = $true)]
        [string] $text = $null
    )

    $filePath = "$env:temp\List.txt"
    $text > "$filePath"
    notepad "$filePath" | out-null
    del "$filePath"
}

function listFiles {
    param(
        [switch] $recurse = $false
    )

    get-childItem -name -recurse:$recurse -force | sort-object $function:sortNaturally | out-string
}

function listTree {
    tree /f
}

function getBufferText {
    $rawUi = $host.ui.rawUi
    $width = [Math]::max([Math]::max($rawUi.bufferSize.width, $rawUi.windowSize.width) - 1, 0)
    $height = [Math]::max($rawUi.cursorPosition.y - 1, 0)

    $lines = new-object System.Text.StringBuilder
    $characters = new-object System.Text.StringBuilder

    for ($h = 0; $h -lt $height; $h += 1) {
        $rectangle = new-object System.Management.Automation.Host.Rectangle 0, $h, $width, $h
        $buffer = $rawUi.getBufferContents($rectangle)

        for ($w = 0; $w -lt $width; $w += 1) {
            $cell = $buffer[0, $w]
            $character = $cell.character
            $characters.append($character) | out-null
        }

        $lines.appendLine($characters.toString()) | out-null
        $characters.length = 0
    }

    $lines.toString() -replace '[ \0]*\r?\n', "`r`n"
}

function main {
    param(
        [parameter(mandatory = $true)]
        [string] $type = $null,

        [parameter(mandatory = $true)]
        [string] $directory = $null
    )

    $outputEncoding = [Text.UTF8Encoding]::UTF8
    [Console]::outputEncoding = [Text.UTF8Encoding]::UTF8
    $PSDefaultParameterValues['out-file:encoding'] = 'utf8'

    set-location -literalPath "$directory"

    $typeFunction = @{
        'files' = { writeList -text $(listFiles) };
        'filesRecursively' = { writeList -text $(listFiles -recurse) };
        'tree' = {
            listTree
            writeList -text $(getBufferText)
        }
    }

    &($typeFunction.get_item($type))
}

main @args

Answer 3

If you output as non-Unicode (which you apparently do), you have to view the text file you create using the same encoding the Console window uses. That's why it looks correct in the console. In some text editors, you can choose an encoding (or "code page") when you open a file. (How to output as Unicode I don't know. cmd /U doesn't do what the documentation says.)

The Console encoding depends on your Windows installation. For me, it's "Western European (DOS)" (or just "MS-DOS") in Microsoft Word.

Answer 4

I decided I had to have a look at tree.com and figure out why it's not respecting the Unicode setting of the console. It turns out that (like many of the command-line file utilities), it uses a library called ulib.dll to do all the printing (specifically, TREE::DisplayName calls WriteString in ulib).

Now, in ulib, the WriteString method is implemented in two classes, SCREEN and STREAM. The SCREEN version uses WriteConsoleW directly, so all the Unicode characters get correctly displayed. The STREAM version converts the Unicode text to one of three different encodings (_UseConsoleConversions ⇒ console codepage (GetConsoleCP), _UseAnsiConversions ⇒ default ANSI codepage, otherwise ⇒ default OEM codepage), and then writes this out. I don't know how to change the conversion mode, and I don't believe the conversion can be disabled.

I've only looked at this briefly, so perhaps more adventurous souls can speak more about it! :-)

Answer 5

This will save the results as ASCII (American Standard Code for Information Interchange) on your desktop, ASCII\ANSI doesn't recognize every international or extended character:

tree /f > ascii.txt

This will convert your ASCII text to Unicode (/c must precede actual command):

cmd /u /c type ascii.txt > unicode.txt

So why not just think of the ascii file as a temporary file and delete it?

del ascii.txt

If you must put all in one line you could use:

tree /f > ascii.txt & cmd.exe /u /c type ascii.txt > unicode.txt & del ascii.txt

Answer 6

The short answer is you cannot and this is because tree.com is an ANSI application, even on Windows 7.

The only solution is to write your own tree implementation. Also you could file a bug to Microsoft, but I doubt they are not already aware about it.