Printing to file python script running as a background process

Question

I have a python script that takes about 3 hours to run. I want to print some of the data it generates to a .csv file.

I'm on centos and I'm running my script like so:

python my_script.py > output.csv &

I also tried:

python my_script.py &> output.csv 

Nothing prints to output.csv, not even some test statements right at the beginning of the script.

The next thing I tried was opening a file in my script - f = open('output.csv', 'a') and writing to it using f.write(). Again, nothing shows up in the file (although it does get created).

How can I create my output.csv with the data that I want? It seems to work normally when it's not a background process but I'd like it to run in the background.

I'm printing just like so:

print('hello, this is a test')

and in another version, I have something like this:

f = open('output.csv', 'a')
f.write('hello, this is a test')

I get exactly the same result in both cases. The file gets created, but nothing is actually written to it.

Any help is appreciated. Thank you!

Answer 1

Try flushing the stdout buffer:

Python >= 3.3:

print('hello, this is a test', flush=True)

Earlier versions:

import sys

print('hello, this is a test')
sys.stdout.flush()