I have some python code that parse the csv file. Now our vendor decide to change the data file to gzip csv file. I was wondering what's the minimal/cleanest code change I have to make. Current function:
def load_data(fname, cols=()):
... ...
with open(fname) as f:
reader = csv.DictReader(f)
... ...
I don't want to duplicate the code to load_data2(), and change the with statement to, thought it works perfectly.
with gzip.open(fname) as f:
How can I factor out the with statement?
def load_data(fname, cols=()):
... ...
if fname.endswith('.csv.gz'):
with gzip.open(fname) as f:
else:
with open(fname) as f:
reader = csv.DictReader(f)
... ... # code to parse
The with statement in Python is used for resource management and exception handling. You'd most likely find it when working with file streams. For example, the statement ensures that the file stream process doesn't block other processes if an exception is raised, but terminates properly.
Instructions that a Python interpreter can execute are called statements. For example, a = 1 is an assignment statement. if statement, for statement, while statement, etc. are other kinds of statements which will be discussed later.
Opening files using the with statement is generally recommended because it ensures that open file descriptors are automatically closed after the flow of execution leaves the with code block. As you saw before, the most common way to open a file using with is through the built-in open() : with open("hello.
There are two main ways to make a nested if statement. The first option is to put the if statement inside an if code block. The other option is to place the if statement in the else code of an if/else statement. Python evaluates this nested if statement when the condition of the preceding if statement is True .
You can do this by assigning the function you want to use to open the file to a different variable, depending on the properties of the file name:
opener = gzip.open if fname.endswith('.csv.gz') else open
with opener(fname) as f:
... # code to parse
The universal approach is to dynamically choose an opener
:
openers = {
'http': urllib2.urlopen,
'.csv.gz': gzip.open
'.csv': open
}
resource_type = get_resource_type(resource) # determine the type of the resource
with openers[resource_type](resource) as f:
# do stuff ...
That way, you can painlessly add more openers, when needed. Here is another example of the factory method design pattern.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With