Windows7 PowerShell inserts ^@ between every characters - how to prevent?

Question

Hopefully for you this would be trivial to answer.

I have gvim and vim on Win7. Any time I redirect command result to file and open it with Vim (or Gvim, same effect) I have binary 0 (^@ sign) inserted between every characters.

So, instead of

r96130

I have

^@r^@9^@6^@1^@3^@0^@

Any idea how to (preferred) prevent it or (fine) get rid of it?

I tried substitution but obviously this ain't just ^@, it's a binary zero (I think) so it's not picked up on /^@.

Edit(s):

1) Tried setting encoding to UTF-8 (encoding of content I am working on) and UTF-16 (per @Philippe Wendler's answer, thank you), to no avail.

2) __PowerShell__ is the guilty party. When I do the redirection on usual command line (via cmd) then there are no side-effects. Only files created on PowerShell sessions have these binary zeroes inserted.

3) Powershell goodies that @Christian mentioned didn't help. I'll leave the question unanswered as I've switched to Ubuntu as my main OS and use Vim there, so the only 'solution' here is now not to use PowerShell + Vim, which obviously is lacking as an answer. :-)

Solution!

Thanks @Emperor XLII for a solution! Posting here as it was in a comment so might not be visible enough:

dir | OutFile -Encoding UTF8 test 

creates a file that appears in Vim without binary zeroes. Same happens if I run with Encoding to OEM, ASCII, and two others I've tested.

A promising tip lies in :he 'fileencodings' - yes, plural, by @Dan Fitch, but I haven't tested it as I aready moved away from Win7 by that time.

Answer 1

Windows uses UTF-16 internally as Unicode encoding, and this file looks like it also uses UTF-16, but vim interprets it with an 8-bit charset (probably latin1).

Try :set encoding=utf-16 in vim to select the right encoding.

Answer 2

I had the same issue. Using a cmd shell (or Console2) and running a command such as this:

prompt> myBatFile.bat > myOutputFile.log

I would get a bunch of ^@ symbols, which I could get rid of using a macro that just ran the following command:

:s/\r\(\n\)/\1/g

This was acceptable for me, but annoying (I believe that it was a combination of having the different unix vs. Dos-style carriage returns and having no end-of-line at the end of the file). To make matters worse, I started using PowerShell once I moved to Windows 7. Once this happened, not only did I get ^@ symbols, but I also had ^M sprinkled in there along with a ÿþ at the beginning of the file (I think that was a byte order mark that gVIM was not able to interpret properly).

At any rate, I changed how I called my bat file so that it followed the PowerShell way of redirecting and now have no issues reading the file in gVIM:

prompt> myBatFile.bat | Out-File myOutputFile.log -encoding UTF8

Hope that helps.

Answer 3

You shouldn't have to do anything special with Powershell's output. You can change your vim configuration so it can deal with the correct encodings.

Look into :he fileencodings (note the trailing s) -- this setting lets you choose what encodings vim will try. By default, Vim 7.x will use ucs-bom and try to decode the byte order mark, but your vimrc may be setting a different list; make sure it isn't starting with something like latin1. By default, on my x64 Windows 7 machine, Powershell command output piped to a file is read by vim just fine.

Either setting fileencodings to ucs-bom or utf-16 should fix your problem.

Answer 4

In powershell try to redirect in a file using

set-content c:\file.txt -encoding Unicode # or BigEndianUnicode

you can also use out-file in the same manner