Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Python file open/close every time vs keeping it open until the process is finished

Tags:

python

file-io

I have about 50 GB of text file and I am checking the first few characters each line and writing those to other files specified for that beginning text.

For example. my input contains:

cow_ilovecow dog_whreismydog cat_thatcatshouldgotoreddit dog_gotitfromshelter ............... 

So, I want to process them in cow, dog and cat (about 200) categories so,

if writeflag==1:     writefile1=open(writefile,"a") #writefile is somedir/dog.txt....     writefile1.write(remline+"\n")     #writefile1.close() 

so, what is the best way, should I close? Otherwise if I keep it open, is writefile1=open(writefile,"a") doing the right thing?

like image 227
Ananta Avatar asked Jul 05 '12 16:07

Ananta


People also ask

Is it necessary to close open file in Python?

You've learned why it's important to close files in Python. Because files are limited resources managed by the operating system, making sure files are closed after use will protect against hard-to-debug issues like running out of file handles or experiencing corrupted data.

Does Python automatically close open files?

Python automatically closes a file when the reference object of a file is reassigned to another file. It is a good practice to use the close() method to close a file.

Does Python close files automatically on exit?

When your script exits, via normal return, exception or calling os. exit(), python will destroy objects that have gone out of scope. This will likely close the files.

What is the difference between open () and close () in Python?

How is file open() function different from close() function? open() as it name implies used to open a data file in a program through file object and close() is used to close() the link of file object and data file.


2 Answers

You should definitely try to open/close the file as little as possible

Because even comparing with file read/write, file open/close is far more expensive

Consider two code blocks:

f=open('test1.txt', 'w') for i in range(1000):     f.write('\n') f.close() 

and

for i in range(1000):     f=open('test2.txt', 'a')     f.write('\n')     f.close() 

The first one takes 0.025s while the second one takes 0.309s

like image 139
xvatar Avatar answered Sep 17 '22 12:09

xvatar


Use the with statement, it automatically closes the files for you, do all the operations inside the with block, so it'll keep the files open for you and will close the files once you're out of the with block.

with open(inputfile)as f1, open('dog.txt','a') as f2,open('cat.txt') as f3:    #do something here 

EDIT: If you know all the possible filenames to be used before the compilation of your code then using with is a better option and if you don't then you should use your approach but instead of closing the file you can flush the data to the file using writefile1.flush()

like image 22
Ashwini Chaudhary Avatar answered Sep 18 '22 12:09

Ashwini Chaudhary