Performance changes significantly every time I rerun program

Question

I have a Rasberry PI like hardware which has a basic linux operating system and only one program running called "Main_Prog".

Every time I run a performance test again Main_Prog I get a less than 1% performance fluctuation. This is perfectly acceptable.

When I kill Main_Prog using the kill command, and re-start Main_Prog, the performance changes up to 8%. Further performance tests will vary less than 1% around this fluctuation.

---

Example

So for example if Main_Prog at first ran at 100 calls/sec and varied between 99-101 calls/sec.

I then did a "kill" command against Main_Prog and restarted using "./Main_Prog &". I then run a performance test and now Main_Prog is running 105 calls/sec with 104-106 calls/sec fluctuation. It will continue to run 104-106 calls/sec until I kill the Main_Prog and start it.

---

Any idea how to prevent fluctuation or what is happening? Remember, it is VERY consistent. No other programs running on operating system.

Answer 1

Your temporary fluctuation might be related to the page cache. I would not bother (the change is insignificant). See also http://www.linuxatemyram.com/

You might prefill the page cache, e.g. by running some wc Main_Prog before running ./Main_Prog

And you probably still do have some other executable programs & processes on your Raspberry Pi (check with top or ps auxw). I guess that /sbin/init is still running at pid 1. And probably your shell is running too.

^{It is quite unusual to have a Linux system with only one process. To get that, you should replace /sbin/init with your program, and I really don't recommend that, especially if you don't know Linux very well.}

Since there are several processes running in your box, and because the kernel scheduler is preempting tasks at arbitrary moment, its behavior is not entirely reproducible, and that explains the observed fluctuation.

Read also more about real-time scheduling, setpriority(2), sched_setscheduler(2), pthread_setschedparam(3), readahead(2), mlock(2), madvise(2), posix_fadvise(2)

If you are mostly interested in benchmarking, the sensible way is to repeat the same benchmark several times (e.g. 4 to 15 times) and either take the minimum, or the maximum, or the average.