Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Python "with" statement syntax

I have some python code that parse the csv file. Now our vendor decide to change the data file to gzip csv file. I was wondering what's the minimal/cleanest code change I have to make. Current function:

def load_data(fname, cols=()):
    ... ...
    with open(fname) as f:
        reader = csv.DictReader(f)
        ... ... 

I don't want to duplicate the code to load_data2(), and change the with statement to, thought it works perfectly.

with gzip.open(fname) as f:

How can I factor out the with statement?

def load_data(fname, cols=()):
    ... ...
    if fname.endswith('.csv.gz'):
        with gzip.open(fname) as f:
    else:
        with open(fname) as f:

        reader = csv.DictReader(f)
        ... ... # code to parse
like image 923
fivelements Avatar asked Oct 28 '13 15:10

fivelements


People also ask

What is the with statement in Python?

The with statement in Python is used for resource management and exception handling. You'd most likely find it when working with file streams. For example, the statement ensures that the file stream process doesn't block other processes if an exception is raised, but terminates properly.

What are statements and syntax in Python?

Instructions that a Python interpreter can execute are called statements. For example, a = 1 is an assignment statement. if statement, for statement, while statement, etc. are other kinds of statements which will be discussed later.

What is the use of with statement?

Opening files using the with statement is generally recommended because it ensures that open file descriptors are automatically closed after the flow of execution leaves the with code block. As you saw before, the most common way to open a file using with is through the built-in open() : with open("hello.

Can you nest with statements in Python?

There are two main ways to make a nested if statement. The first option is to put the if statement inside an if code block. The other option is to place the if statement in the else code of an if/else statement. Python evaluates this nested if statement when the condition of the preceding if statement is True .


2 Answers

You can do this by assigning the function you want to use to open the file to a different variable, depending on the properties of the file name:

opener = gzip.open if fname.endswith('.csv.gz') else open
with opener(fname) as f:
    ... # code to parse
like image 180
mdml Avatar answered Sep 22 '22 07:09

mdml


The universal approach is to dynamically choose an opener:

openers = {
    'http': urllib2.urlopen,
    '.csv.gz': gzip.open
    '.csv': open
}

resource_type = get_resource_type(resource) # determine the type of the resource

with openers[resource_type](resource) as f:
    # do stuff ...

That way, you can painlessly add more openers, when needed. Here is another example of the factory method design pattern.

like image 28
Alexander Zhukov Avatar answered Sep 22 '22 07:09

Alexander Zhukov