Combining inplace filtering and the setting of encoding in the fileinput module

Question

I am attempting to use the fileinput module's inplace filtering feature to rewrite an input file in place.

Needed to set encoding (both for read and write) to latin-1 and attempted to pass openhook=fileinput.hook_encoded('latin-1') to fileinput.input but was thwarted by the error

ValueError: FileInput cannot use an opening hook in inplace mode

Upon closer inspection I see that the fileinput documentation clearly states this: You cannot use inplace and openhook together

How can I get around this?

Answer 1

Starting python 3.10 fileinput.input() accepts an encoding parameter

Answer 2

As far as I know, there is no way around this with the fileinput module. You can accomplish the same task with a combination of the codecs module, os.rename(), and os.remove():

import os
import codecs

input_name = 'some_file.txt'
tmp_name = 'tmp.txt'

with codecs.open(input_name, 'r', encoding='latin-1') as fi, \
     codecs.open(tmp_name, 'w', encoding='latin-1') as fo:

    for line in fi:
        new_line = do_processing(line) # do your line processing here
        fo.write(new_line)

os.remove(input_name) # remove original
os.rename(tmp_name, input_name) # rename temp to original name

You also have the option of specifying a new encoding for the output file if you want to change it, or leave it as latin-1 when opening the output file if you don't want it it to change.

I know this isn't the in-place modification you were looking for, but it will accomplish the task you were trying to do and is very flexible.

Answer 3

If you don't mind using a pip library, the in_place library supports encoding.

import in_place

with in_place.InPlace(filename, encoding="utf-8") as fp:
  for line in fp:
    fp.write(line)