How to lock a directory between python processes in linux?

Question

I have two (or more) python processes running and want to create a concept similar to an exclusion mutex for a shared resource. The 'shared resource' in this case is a directory. How might I most easily/standardly/etc implement a mutex? A hidden .lock file that each process agrees to check and, if exists, appends their PID as a new row and then pops their PID when they have access to the file?

I basically just want to clear a directory and make sure no other process tries to read or write to it while I'm clearing it.

Is there a standard linux way of doing this? Maybe something I can just execute with a shell line from python?

Answer 1

Linux

There are two standard types of locking in Linux: advisory locking (specified in POSIX) and mandatory locking (Linux-specific).

However, both of them may be applied only on files, but not directories. So yes, you need a lock-file. It assumes that all users should know about the lock-file and acquire the lock before accessing directory. Hence, mandatory locking will not help here, and you need advisory locking.

There are three kinds of advisory file locks in Linux:

• flock(2) (specified in POSIX);
• POSIX record locks, see "Advisory record locking" section in fcntl(2) and also lockf(3) wrapper (both specified in POSIX);
• Open file description locks, see fcntl(2) (Linux-specific, available in recent kernels).

---

Python

In Python, flock(), lockf() and fcntl() functions are available through fcntl module. There is also flock module that adds context-manager support to fcntl.flock function.

Here is an example:

import flock

with open('/my/dir/lockfile', 'w') as fp:
    with flock.Flock(fp, flock.LOCK_EX) as lock:
        pass # exclusive lock is acquired here

---

PS.

With this approach, you can not prevent a random process from accessing your directory, if it don't know about your lock-file. It is probably possible to implement a filesystem using FUSE that will support mandatory directory locks, but I'm not aware of such implementations.

Answer 2

You can apply a flock lock on a directory. This is advisory (it won't stop processes that don't care about the lock touching the directory). The lock will persist until your program terminates. The code is pretty trivial.

lockfd = os.open('.',os.O_RDONLY)
fcntl.flock(lockfd,fcntl.LOCK_EX | fcntl.LOCK_NB)

If my understanding of the manpage is correct you can unlock it by simply doing.

os.close(lockfd)

But I haven't tested this as in my application it's desired for the lock to persist for the scripts entire run.