After I read from a file then write it out again, I am getting Chinese characters

Question

I am reading all the lines in a file and then writing them out again. When I do this, the file I have written out ends up being mostly Chinese characters. I am not modifying any of the lines at all. This is my Python code:

#test.py
import os, sys, time
import getopt

if __name__=='__main__':
    testFile = None
    try:
        optlist, args = getopt.getopt(sys.argv[1:],"",["file="])
    except Exception, e:
        print e
        sys.exit(1)
    for opt, arg in optlist:
        if opt == '--file':
            testFile = arg.strip()

    print testFile
    if os.path.isfile(testFile):
        f = open(testFile, 'r')
        lines = f.readlines()
        f.close()
        f = open(testFile, 'w')
        for line in lines:
            f.write(line)
        f.close()

This is the original file I am testing this code on:

param($Identity = "")


if($_INITIALIZATION_isLoaded -ne $true){
    #load initialization script
    . ((split-path -parent $myInvocation.InvocationName) + "\stuff.ps1")
}

After my file is written, this is the contents of the file:

param($Identity = "")

਀ഀഊ

਀椀昀⠀␀开䤀一䤀吀䤀䄀䰀䤀娀䄀吀䤀伀一开椀猀䰀漀愀搀攀搀 ⴀ渀攀 ␀琀爀甀攀⤀笀ഀഊ #load initialization script

਀ऀ⸀ ⠀⠀猀瀀氀椀琀ⴀ瀀愀琀栀 ⴀ瀀愀爀攀渀琀 ␀洀礀䤀渀瘀漀挀愀琀椀漀渀⸀䤀渀瘀漀挀愀琀椀漀渀一愀洀攀⤀ ⬀ ∀尀猀琀甀昀昀⸀瀀猀㄀∀⤀ഀഊ}

My command line statement to run my Python script is:

python test.py --file="test.txt"

I am doing this with Python 2.7 on Windows 7. What is causing this and how do I fix it? Thanks.

Edit:

If I add a print lines in my script, I get this:

['\xfe\xff\x00p\x00a\x00r\x00a\x00m\x00(\x00$\x00I\x00d\x00e\x00n\x00t\x00i\x00t\x00y\x00 \x00=\x00 \x00"\x00"\x00)\x00\r\x00\n', '\x00\r\x00\n', '\x00\r\x00\n'
, '\x00i\x00f\x00(\x00$\x00_\x00I\x00N\x00I\x00T\x00I\x00A\x00L\x00I\x00Z\x00A\x00T\x00I\x00O\x00N\x00_\x00i\x00s\x00L\x00o\x00a\x00d\x00e\x00d\x00 \x00-\x00n\x
00e\x00 \x00$\x00t\x00r\x00u\x00e\x00)\x00{\x00\r\x00\n', '\x00\t\x00#\x00l\x00o\x00a\x00d\x00 \x00i\x00n\x00i\x00t\x00i\x00a\x00l\x00i\x00z\x00a\x00t\x00i\x00o
\x00n\x00 \x00s\x00c\x00r\x00i\x00p\x00t\x00\r\x00\n', '\x00\t\x00.\x00 \x00(\x00(\x00s\x00p\x00l\x00i\x00t\x00-\x00p\x00a\x00t\x00h\x00 \x00-\x00p\x00a\x00r\x0
0e\x00n\x00t\x00 \x00$\x00m\x00y\x00I\x00n\x00v\x00o\x00c\x00a\x00t\x00i\x00o\x00n\x00.\x00I\x00n\x00v\x00o\x00c\x00a\x00t\x00i\x00o\x00n\x00N\x00a\x00m\x00e\x0
0)\x00 \x00+\x00 \x00"\x00\\\x00s\x00t\x00u\x00f\x00f\x00.\x00p\x00s\x001\x00"\x00)\x00\r\x00\n', '\x00}\x00\r\x00\n']

Answer 1

Your original file is in UTF-16 and a byte has been dropped somewhere, resulting in all your characters being off by one byte.

$ charinfo "䤀一䤀吀"
U+4900 CJK UNIFIED IDEOGRAPH-4900
U+4E00 CJK UNIFIED IDEOGRAPH-4E00
U+4900 CJK UNIFIED IDEOGRAPH-4900
U+5400 CJK UNIFIED IDEOGRAPH-5400
$ charinfo "INIT"
U+0049 LATIN CAPITAL LETTER I
U+004E LATIN CAPITAL LETTER N
U+0049 LATIN CAPITAL LETTER I
U+0054 LATIN CAPITAL LETTER T

Consider using codecs.open() so that it handles transcoding issues.