Linux, timerfd accuracy

Question

I have a system that needs at least 10 mseconds of accuracy for timers.
I went for timerfd as it suits me perfectly, but found that even for times up to 15 milliseconds it is not accurate at all, either that or I don't understand how it works.

The times I have measured were up to 21 mseconds on a 10 mseconds timer.
I have put together a quick test that shows my problem.
Here a test:

#include <sys/timerfd.h>
#include <time.h>
#include <string.h>
#include <stdint.h>
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <inttypes.h>

int main(int argc, char *argv[]){

    int timerfd = timerfd_create(CLOCK_MONOTONIC,0);
    int milliseconds = atoi(argv[1]);
    struct itimerspec timspec;
    bzero(&timspec, sizeof(timspec));
    timspec.it_interval.tv_sec = 0;
    timspec.it_interval.tv_nsec = milliseconds * 1000000;
    timspec.it_value.tv_sec = 0;
    timspec.it_value.tv_nsec = 1;

    int res = timerfd_settime(timerfd, 0, &timspec, 0);
    if(res < 0){
       perror("timerfd_settime:");
       return 1;
    }
    uint64_t expirations = 0;
    int iterations = 0;
    while( res = read(timerfd, &expirations, sizeof(expirations))){
        if(res < 0){ perror("read:"); continue; }
        if(expirations > 1){
            printf("%" PRIu64 " expirations, %d iterations\n", expirations, iterations);
            break;
        }
        iterations++;
    }
    return 0;
}

And executed like this:

Zack ~$ for i in 2 4 8 10 15; do echo "intervals of $i milliseconds"; ./test $i;done
intervals of 2 milliseconds
2 expirations, 1 iterations
intervals of 4 milliseconds
2 expirations, 6381 iterations
intervals of 8 milliseconds
2 expirations, 21764 iterations
intervals of 10 milliseconds
2 expirations, 1089 iterations
intervals of 15 milliseconds
2 expirations, 3085 iterations

Even assuming some possible delays, 15 milliseconds delays sounds too much for me.

Answer 1

Try altering it as follows, this should pretty much garuntee that it'll never miss a wakeup, but be careful with it since running realtime priority can lock your machine hard if it doesn't sleep, also you may need to set things up so that your user has the ability to run stuff at realtime priority (see /etc/security/limits.conf)

#include <sys/timerfd.h>
#include <time.h>
#include <string.h>
#include <stdint.h>
#include <stdio.h>
#include <sched.h>

int main(int argc, char *argv[]) 
{
    int timerfd = timerfd_create(CLOCK_MONOTONIC,0);
    int milliseconds = atoi(argv[1]);
    struct itimerspec timspec;
    struct sched_param schedparm;

    memset(&schedparm, 0, sizeof(schedparm));
    schedparm.sched_priority = 1; // lowest rt priority
    sched_setscheduler(0, SCHED_FIFO, &schedparm);

    bzero(&timspec, sizeof(timspec));
    timspec.it_interval.tv_sec = 0;
    timspec.it_interval.tv_nsec = milliseconds * 1000000;
    timspec.it_value.tv_sec = 0;
    timspec.it_value.tv_nsec = 1;

    int res = timerfd_settime(timerfd, 0, &timspec, 0);
    if(res < 0){
       perror("timerfd_settime:");
    }
    uint64_t expirations = 0;
    int iterations = 0;
    while( res = read(timerfd, &expirations, sizeof(expirations))){
        if(res < 0){ perror("read:"); continue; }
        if(expirations > 1){
            printf("%ld expirations, %d iterations\n", expirations, iterations);
            break;
        }
        iterations++;
    }
}

If you are using threads you should use pthread_setschedparam instead of sched_setscheduler.

Realtime also isn't about low latency, it's about guarantees, RT means that if you want to wake up exactly once every second on the second, you WILL, the normal scheduling does not give you this, it might decide to wake you up 100ms later, because it had some other work to do at that time anyway. If you want to wake up every 10ms and you REALLY do need to, then you should set yourself to run as a realtime task then the kernel will wake you up every 10ms without fail. Unless a higher priority realtime task is busy doing stuff.

If you need to guarantee that your wakeup interval is exactly some time it doesn't matter if it's 1ms or 1 second, you won't get it unless you run as a realtime task. There are good reasons the kernel will do this to you (saving power is one of them, higher throughput is another, there are others), but it's well within it's rights to do so since you never told it you need better guarantees. Most stuff doesn't actually need to be this accurate, or need to never miss so you should think hard about whether or not you really do need it.

quoting from http://www.ganssle.com/articles/realtime.htm

A hard real time task or system is one where an activity simply must be completed - always - by a specified deadline. The deadline may be a particular time or time interval, or may be the arrival of some event. Hard real time tasks fail, by definition, if they miss such a deadline.

Notice this definition makes no assumptions about the frequency or period of the tasks. A microsecond or a week - if missing the deadline induces failure, then the task has hard real time requirements.

Soft realtime is pretty much the same, except that missing a deadline, while undesirable, is not the end of the world (for example video and audio playback are soft realtime tasks, you don't want to miss displaying a frame, or run out of buffer, but if you do it's just a momentary hiccough, and you simply continue). If what you are trying to do is 'soft' realtime I wouldn't bother with running at realtime priority since you should generally get your wakeups in time (or at least close to it).

EDIT:

If you aren't running realtime the kernel will by default give any timers you make some 'slack' so that it can merge your request to wake up with other events that happen at times close to the one you asked for (that is if the other event is within your 'slack' time it will not wake you at the time you asked, but a little earlier or later, at the same time it was already going to do something else, this saves power).

For a little more info see High- (but not too high-) resolution timeouts and Timer slack (note I'm not sure if either of those things is exactly what's really in the kernel since both those articles are about lkml mailing list discussions, but something like the first one really is in the kernel.

Answer 2

I've got a feeling that your test is very hardware dependent. When I ran your sample program on my system, it appeared to hang at 1ms. To make your test at all meaningful on my computer, I had to change from milliseconds to microseconds. (I changed the multiplier from 1_000_000 to 1_000.)

$ grep 1000 test.c
    timspec.it_interval.tv_nsec = microseconds * 1000;

$ for i in 1 2 4 5 7 8 9 15 16 17\
 31 32 33 47 48 49 63 64 65 ; do\
 echo "intervals of $i microseconds";\
 ./test $i;done
intervals of 1 microseconds
11 expirations, 0 iterations
intervals of 2 microseconds
5 expirations, 0 iterations
intervals of 4 microseconds
3 expirations, 0 iterations
intervals of 5 microseconds
2 expirations, 0 iterations
intervals of 7 microseconds
2 expirations, 0 iterations
intervals of 8 microseconds
2 expirations, 0 iterations
intervals of 9 microseconds
2 expirations, 0 iterations
intervals of 15 microseconds
2 expirations, 7788 iterations
intervals of 16 microseconds
4 expirations, 1646767 iterations
intervals of 17 microseconds
2 expirations, 597 iterations
intervals of 31 microseconds
2 expirations, 370969 iterations
intervals of 32 microseconds
2 expirations, 163167 iterations
intervals of 33 microseconds
2 expirations, 3267 iterations
intervals of 47 microseconds
2 expirations, 1913584 iterations
intervals of 48 microseconds
2 expirations, 31 iterations
intervals of 49 microseconds
2 expirations, 17852 iterations
intervals of 63 microseconds
2 expirations, 24 iterations
intervals of 64 microseconds
2 expirations, 2888 iterations
intervals of 65 microseconds
2 expirations, 37668 iterations

(Somewhat interesting that I got the longest runs from 16 and 47 microseconds, but 17 and 48 were awful.)

time(7) has some suggestions on why our platforms are so different:

   High-Resolution Timers
       Before Linux 2.6.21, the accuracy of timer and sleep system
       calls (see below) was also limited by the size of the jiffy.

       Since Linux 2.6.21, Linux supports high-resolution timers
       (HRTs), optionally configurable via CONFIG_HIGH_RES_TIMERS.  On
       a system that supports HRTs, the accuracy of sleep and timer
       system calls is no longer constrained by the jiffy, but instead
       can be as accurate as the hardware allows (microsecond accuracy
       is typical of modern hardware).  You can determine whether
       high-resolution timers are supported by checking the resolution
       returned by a call to clock_getres(2) or looking at the
       "resolution" entries in /proc/timer_list.

       HRTs are not supported on all hardware architectures.  (Support
       is provided on x86, arm, and powerpc, among others.)

All the 'resolution' lines in my /proc/timer_list are 1ns on my (admittedly ridiculously powerful) x86_64 system.

I decided to try to figure out where the 'breaking point' is on my computer, but gave up on the 110 microsecond run:

$ for i in 70 80 90 100 110 120 130\
 ; do echo "intervals of $i microseconds";\
 ./test $i;done
intervals of 70 microseconds
2 expirations, 639236 iterations
intervals of 80 microseconds
2 expirations, 150304 iterations
intervals of 90 microseconds
4 expirations, 3368248 iterations
intervals of 100 microseconds
4 expirations, 1964857 iterations
intervals of 110 microseconds
^C

90 microseconds ran for three million iterations before it failed a few times; that's 22 times better resolution than your very first test, so I'd say that given the right hardware, 10ms shouldn't be anywhere near difficult. (90 microseconds is 111 times better resolution than 10 milliseconds.)

But if your hardware doesn't provide the timers for high resolution timers, then Linux can't help you without resorting to SCHED_RR or SCHED_FIFO. And even then, perhaps another kernel could better provide you with the software timer support you need.

Good luck. :)