Opening already opened file does not raise exception

Question

Consider those two python programs:

script_a.py:

from datetime import datetime
from time import sleep

while True:
    sleep(1)
    with open('foo.txt', 'w') as f:
        sleep(3)
        s = str(datetime.now())
        f.write(s)
        sleep(3)

script_b.py:

while True:
    with open('foo.txt') as f:
        s = f.read()
        print s

Run script_a.py. While it is running, start script_b.py. Both will happily run, but script_b.py outputs an empty string if the file is currently opened by script_a.py.

I was expecting an IOError exception to be raised, telling me the file is already opened, but it didn't happen, instead the file looks empty. Why is that and what would be the proper way to check if it is opened by another process? Would it be ok to simply check if an empty string is returned and try again until something else is read, or is there a more pythonic way?

Answer 1

See the other answer and comments regarding how multiple file opens work in Python. If you've read all that, and still want to lock access to the file on a POSIX platform, then you can use the fcntl library.

Keep in mind that: A) other programs may ignore your lock on the file, B) some networked file systems don't implement locking very well, or at all C) be sure to be very careful to release locks and avoid deadlock as flock won't detect it [1][2].

Example.... script_a.py

from datetime import datetime
from time import sleep
import fcntl

while True:
    sleep(1)
    with open('foo.txt', 'w') as f:
        s = str(datetime.now())

        print datetime.now(), "Waiting for lock"
        fcntl.flock(f, fcntl.LOCK_EX)
        print datetime.now(), "Lock clear, writing"

        sleep(3)
        f.write(s)

        print datetime.now(), "releasing lock"
        fcntl.flock(f, fcntl.LOCK_UN)

script_b.py

import fcntl
from datetime import datetime

while True:
    with open('foo.txt') as f:
        print datetime.now(), "Getting lock"
        fcntl.flock(f, fcntl.LOCK_EX)
        print datetime.now(), "Got lock, reading file"

        s = f.read()

        print datetime.now(), "Read file, releasing lock"
        fcntl.flock(f, fcntl.LOCK_UN)

        print s

Hope this helps!

Answer 2

You are allowed to open a file as many times as you want, so long as the operating system doesn't stop you. This is occasionally useful to get multiple cursors into a file for complex operations.

The reason that script_b.py thinks that the file is empty is that the file is empty:

with open('foo.txt', 'w') as f:

opening a file in w mode immediately erases (i.e. truncates) the file. There's an initial three second gap in script_a where the file is completely 100% empty, and that's what script_b sees.

In the next three second gap after you call f.write, the file is still... probably empty. This is due to buffering - the file on disk is not guaranteed to contain everything that you have written to it with write until you either close (i.e. exit the context manager block) or manually invoke flush on the file handle.

Alternatively, you can open in unbuffered mode, so that writes are always immediately written to disk.

with open('foo.txt','w',0) as f:
   #no buffering, f.write() writes immediately to disk