Ruby file handle management (too many open files)

Question

I am performing very rapid file access in ruby (2.0.0 p39474), and keep getting the exception Too many open files

Having looked at this thread, here, and various other sources, I'm well aware of the OS limits (set to 1024 on my system).

The part of my code that performs this file access is mutexed, and takes the form:

File.open( filename, 'w'){|f| Marshal.dump(value, f) }

where filename is subject to rapid change, depending on the thread calling the section. It's my understanding that this form relinquishes its file handle after the block.

I can verify the number of File objects that are open using ObjectSpace.each_object(File). This reports that there are up to 100 resident in memory, but only one is ever open, as expected.

Further, the exception itself is thrown at a time when there are only 10-40 File objects reported by ObjectSpace. Further, manually garbage collecting fails to improve any of these counts, as does slowing down my script by inserting sleep calls.

My question is, therefore:

• Am I fundamentally misunderstanding the nature of the OS limit---does it cover the whole lifetime of a process?
  • If so, how do web servers avoid crashing out after accessing over ulimit -n files?
  • Is ruby retaining its file handles outside of its object system, or is the kernel simply very slow at counting 'concurrent' access?

Edit 20130417: strace indicates that ruby doesn't write all of its data to the file, returning and releasing the mutex before doing so. As such, the file handles stack up until the OS limit.

In an attempt to fix this, I have used syswrite/sysread, synchronous mode, and called flush before close. None of these methods worked.

My question is thus revised to: Why is ruby failing to close its file handles, and how can I force it to do so?

Answer 1

Use dtrace or strace or whatever equivalent is on your system, and find out exactly what files are being opened.

Note that these could be sockets.

I agree that the code you have pasted does not seem to be capable of causing this problem, at least, not without a rather strange concurrency bug as well.