Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Convert UTF-8 with BOM to UTF-8 with no BOM in Python

Two questions here. I have a set of files which are usually UTF-8 with BOM. I'd like to convert them (ideally in place) to UTF-8 with no BOM. It seems like codecs.StreamRecoder(stream, encode, decode, Reader, Writer, errors) would handle this. But I don't really see any good examples on usage. Would this be the best way to handle this?

source files: Tue Jan 17$ file brh-m-157.json  brh-m-157.json: UTF-8 Unicode (with BOM) text 

Also, it would be ideal if we could handle different input encoding wihtout explicitly knowing (seen ASCII and UTF-16). It seems like this should all be feasible. Is there a solution that can take any known Python encoding and output as UTF-8 without BOM?

edit 1 proposed sol'n from below (thanks!)

fp = open('brh-m-157.json','rw') s = fp.read() u = s.decode('utf-8-sig') s = u.encode('utf-8') print fp.encoding   fp.write(s) 

This gives me the following error:

IOError: [Errno 9] Bad file descriptor 

Newsflash

I'm being told in comments that the mistake is I open the file with mode 'rw' instead of 'r+'/'r+b', so I should eventually re-edit my question and remove the solved part.

like image 427
timpone Avatar asked Jan 17 '12 16:01

timpone


People also ask

How do I remove a BOM from UTF-8 in Python?

Set the encoding to utf-8-sig to remove the BOM character when reading from a file, e.g. with open('example. txt', 'r', encoding='utf-8-sig') as f: . The utf-8--sig encoding skips the BOM byte if it appears as the first byte in the file.

What is UTF-8 without BOM?

The UTF-8 encoding without a BOM has the property that a document which contains only characters from the US-ASCII range is encoded byte-for-byte the same way as the same document encoded using the US-ASCII encoding. Such a document can be processed and understood when encoded either as UTF-8 or as US-ASCII.


2 Answers

Simply use the "utf-8-sig" codec:

fp = open("file.txt") s = fp.read() u = s.decode("utf-8-sig") 

That gives you a unicode string without the BOM. You can then use

s = u.encode("utf-8") 

to get a normal UTF-8 encoded string back in s. If your files are big, then you should avoid reading them all into memory. The BOM is simply three bytes at the beginning of the file, so you can use this code to strip them out of the file:

import os, sys, codecs  BUFSIZE = 4096 BOMLEN = len(codecs.BOM_UTF8)  path = sys.argv[1] with open(path, "r+b") as fp:     chunk = fp.read(BUFSIZE)     if chunk.startswith(codecs.BOM_UTF8):         i = 0         chunk = chunk[BOMLEN:]         while chunk:             fp.seek(i)             fp.write(chunk)             i += len(chunk)             fp.seek(BOMLEN, os.SEEK_CUR)             chunk = fp.read(BUFSIZE)         fp.seek(-BOMLEN, os.SEEK_CUR)         fp.truncate() 

It opens the file, reads a chunk, and writes it out to the file 3 bytes earlier than where it read it. The file is rewritten in-place. As easier solution is to write the shorter file to a new file like newtover's answer. That would be simpler, but use twice the disk space for a short period.

As for guessing the encoding, then you can just loop through the encoding from most to least specific:

def decode(s):     for encoding in "utf-8-sig", "utf-16":         try:             return s.decode(encoding)         except UnicodeDecodeError:             continue     return s.decode("latin-1") # will always work 

An UTF-16 encoded file wont decode as UTF-8, so we try with UTF-8 first. If that fails, then we try with UTF-16. Finally, we use Latin-1 — this will always work since all 256 bytes are legal values in Latin-1. You may want to return None instead in this case since it's really a fallback and your code might want to handle this more carefully (if it can).

like image 83
Martin Geisler Avatar answered Oct 05 '22 23:10

Martin Geisler


In Python 3 it's quite easy: read the file and rewrite it with utf-8 encoding:

s = open(bom_file, mode='r', encoding='utf-8-sig').read() open(bom_file, mode='w', encoding='utf-8').write(s) 
like image 22
Geng Jiawen Avatar answered Oct 05 '22 23:10

Geng Jiawen