Return file handles opened with with open?

Question

I'm creating software where I want to accept compressed files. Since files are read/written everywhere, I created a utility function for opening files, that handles opening/closing for me, for some compressed filetypes.

Example code:

def return_file_handle(input_file, open_mode="r"):
    """ Handles compressed and uncompressed files. Accepts open modes r/w/w+ """

    if input_file.endswith(".gz")
        with gzip.open(input_file, open_mode) as gzipped_file_handle:
            return gzipped_file_handle

The problem is, when using this code, the file handle seems to close when the function returns. I it possible to do what I want with with open or do I need to handle closing myself?

Add this to the code above to get a minimal non-working example:

for line in return_file_handle(input_bed, "rb"):
    print line

Create a gzipped textfile with:

echo "hei\nder!" | gzip - > test.gz

Error message:

Traceback (most recent call last):
  File "check_bed_against_blacklist.py", line 26, in <module>
    check_bed_against_blacklist("test.gz", "bla")
  File "check_bed_against_blacklist.py", line 15, in check_bed_against_blacklist
    for line in return_file_handle(input_bed, "r"):
ValueError: I/O operation on closed file.

Answer 1

Try it as a generator:

def return_file_handle(input_file, open_mode="r"):
    """
    Handles compressed and uncompressed files. Accepts open modes r/w/w+
    """
    # compressed
    if input_file.endswith(".gz"):
        with gzip.open(input_file, open_mode) as gzipped_file_handle:
            yield gzipped_file_handle
    else:
        with open(input_file, open_mode) as normal_fh:
            yield normal_fh

When you call it:

for line in return_file_handle("file.gz"):
    print(line.read())

Or composing a generator using python's new yield from syntax:

def each_line(fh):
    for l in fh:
        yield from l

And calling it:

for each in each_line(return_file_handle(fh)):
    print(each)

with the file cleanly closing at the end of the for loop.

Answer 2

The best way I can think of is passing a function as parameter which accept opened fd:

def work(fd):
    for line in fd:
        print line

def work_with_file_handle(input_file, func, open_mode="r"):
   if input_file.endswith(".gz")
       with gzip.open(input_file, open_mode) as gzipped_file_handle:
           func(gzipped_file_handle)

work_with_file_handle('xxx.gz', work)

Answer 3

Avoid with, if you want to return the file_handle. Because file_handle will be automatically closed when with block is finished executing.

Following code is what you should be using:

import gzip
def return_file_handle(input_file, open_mode="rb"):
    if input_file.endswith(".gz"):
        gzipped_file_handle = gzip.open(input_file, open_mode)
        return gzipped_file_handle

for line in return_file_handle('file.txt.gz', "r"):
    print line