IOError: close() called during concurrent operation on the same file object

Question

What are the common causes of this error? (w/o looking at the code)

I've googled it but didnt seem to any reference for this type of error.

My initial guess is that (by looking at the error msg), there is a thr/proc doing something to the object while another thr/proc is trying to close it.

Is there a lock-like mechanism for file object?

Answer 1

Some parts of file-object code allow multi-threading -- they release GIL during operations, i.e. file_read() in python-2.7.9/Objects/fileobject.c contains that lines:

FILE_BEGIN_ALLOW_THREADS(f)
errno = 0;
chunksize = Py_UniversalNewlineFread(BUF(v) + bytesread,
            buffersize - bytesread, f->f_fp, (PyObject *)f);
interrupted = ferror(f->f_fp) && errno == EINTR;
FILE_END_ALLOW_THREADS(f)

FILE_BEGIN_ALLOW_THREADS increases reference counter, which is checked on close()-side:

/*
* These macros release the GIL while preventing the f_close() function being
* called in the interval between them.  For that purpose, a running total of
* the number of currently running unlocked code sections is kept in
* the unlocked_count field of the PyFileObject. The close() method raises
* an IOError if that field is non-zero.  See issue #815646, #595601.
*/

#define FILE_BEGIN_ALLOW_THREADS(fobj) \
{ \
    fobj->unlocked_count++; \
    Py_BEGIN_ALLOW_THREADS

I tried to reproduce such behavior with following script, but it segfaults my interpreter:

import sys
from threading import Thread

def read():
    sys.stdin.read(10)

t = Thread(target = read)
t.start()
sys.stdin.close()
t.join()

Anyway, you can use Lock objects from threading module:

import sys
from threading import Thread, Lock

stdin_lock = Lock()

def read():
    with stdin_lock:
        sys.stdin.read(10)

t = Thread(target = read)
t.start()
with stdin_lock:
    sys.stdin.close()
t.join()