Are epoll events being watched when not epoll_waiting

Question

I'm rather new to event based programming. I'm experimenting with epoll's edge-mode which apparently only signals files which have become ready for read/write (as opposed to level-mode which signals all ready files, regardless of whether there were already ready, or just became ready).

What's not clear to me, is: in edge-mode, am I informed of readiness events that happen while I'm not epoll_waiting ? What about events on one-shot files that haven't been rearmed yet ?

To illustrate why I'm asking that, consider the following scenario:

• have 10 non-blocking sockets connected
• configure epoll_ctl to react when the sockets are ready for read, in edge-mode + oneshot : EPOLLET | EPOLLONESHOT | EPOLLIN
• epoll_wait for something to happen (reports max 10 events)
• linux wakes my process and reports sockets #1 and #2 are ready
• I read and process data socket #1 (until E_AGAIN)
• I read and process data socket #2 (until E_AGAIN)
• While I'm doing that, a socket S receives data
• I processed all events, so I rearm the triggered files with epoll_ctl in EPOLL_CTL_MOD mode, because of oneshot
• my loop goes back to epoll_waiting the next batch of events

Ok, so will the last epoll_wait always be notified of the readiness of socket S ? Event if S is #1 (i.e. it's not rearmed) ?

Answer 1

I'm experimenting with epoll's edge-mode which apparently only signals files which have become ready for read/write (as opposed to level-mode which signals all ready files, regardless of whether there were already ready, or just became ready)

First let's get a clear view of the system, you need an accurate mental model of how the system works. Your view of epoll(7) is not really accurate.

The difference between edge-triggered and level-triggered is the definition of what exactly makes an event. The former generates one event for each action that has been subscribed on the file descriptor; once you consume the event, it is gone - even if you didn't consume all the data that generated such an event. OTOH, the latter keeps generating the same event over and over until you consume all the data that generated the event.

Here's an example that puts these concepts in action, blatantly stolen from man 7 epoll:

1. The file descriptor that represents the read side of a pipe (rfd) is registered on the epoll instance.

2. A pipe writer writes 2 kB of data on the write side of the pipe.

3. A call to epoll_wait(2) is done that will return rfd as a ready file descriptor.

4. The pipe reader reads 1 kB of data from rfd.

5. A call to epoll_wait(2) is done.

If the rfd file descriptor has been added to the epoll interface using the EPOLLET (edge-triggered) flag, the call to epoll_wait(2) done in step 5 will probably hang despite the available data still present in the file input buffer; meanwhile the remote peer might be expecting a response based on the data it already sent. The reason for this is that edge-triggered mode delivers events only when changes occur on the monitored file descriptor. So, in step 5 the caller might end up waiting for some data that is already present inside the input buffer. In the above example, an event on rfd will be generated because of the write done in 2 and the event is consumed in 3. Since the read operation done in 4 does not consume the whole buffer data, the call to epoll_wait(2) done in step 5 might block indefinitely.

In short, the fundamental difference is in the definition of "event": edge-triggered treats events as a single unit that you consume once; level-triggered defines the consumption of an event as being equivalent to consuming all of the data belonging to that event.

Now, with that out of the way, let's address your specific questions.

in edge-mode, am I informed of readiness events that happen while I'm not epoll_waiting

Yes, you are. Internally, the kernel queues up the interesting events that happened on each file descriptor. They are returned on the next call to epoll_wait(2), so you can rest assured that you won't lose events. Well, maybe not exactly on the next call if there are other events pending and the events buffer passed to epoll_wait(2) can't accommodate them all, but the point is, eventually these events will be reported.

What about events on one-shot files that haven't been rearmed yet?

Again, you never lose events. If the file descriptor hasn't been rearmed yet, should any interesting event arise, it is simply queued in memory until the file descriptor is rearmed. Once it is rearmed, any pending events - including those that happened before the descriptor was rearmed - will be reported in the next call to epoll_wait(2) (again, maybe not exactly the next one, but they will be reported). In other words, EPOLLONESHOT does not disable event monitoring, it simply disables event notification temporarily.

Ok, so will the last epoll_wait always be notified of the readiness of socket S? Event if S is #1 (i.e. it's not rearmed)?

Given what I said above, by now it should be pretty clear: yes, it will. You won't lose any event. epoll offers strong guarantees, it's awesome. It's also thread-safe and you can wait on the same epoll fd in different threads and update event subscription concurrently. epoll is very powerful, and it is well worth taking the time to learn it!