Python: suggestion how to improve to write in streaming text file in Python

Question

I am studying how to write in streaming strings as files in python.

normally i use an expression as

myfile = open("test.txt", w)
for line in mydata:
...     myfile.write(line + '\n')
myfile.close()

Python creates a text file in the directory and save the values chunk-by-chunk at intervals of time.

I have the following questions:

is it possible to set a buffer? (ex: save the data every 20 MB) is it possible to save line-by-line?

thanks for suggestions, they help always to improve

Answer 1

File I/O in python is already buffered. The open() function lets you determine to what extend writing is buffered:

The optional buffering argument specifies the file’s desired buffer size: 0 means unbuffered, 1 means line buffered, any other positive value means use a buffer of (approximately) that size. A negative buffering means to use the system default, which is usually line buffered for tty devices and fully buffered for other files. If omitted, the system default is used.

Personally, I'd use the file as a context manager through the with statement. As soon as all the statements under the with suite (at least one indentation level deeper) have completed or an exception is raised the file object is closed:

with open("test.txt", 'w', buffering=20*(1024**2)) as myfile:
    for line in mydata:
        myfile.write(line + '\n')

where I've set the buffer to 20 MB in the above example.