Why does my CPU suddenly work twice as fast?

Question

I've been trying to use a simple profiler to measure the efficiency of some C code on a school server, and I'm hitting an odd situation. After a short amount of time (half a second-ish), the processor suddenly starts executing instructions twice as fast. I've tested for just about every possible reason I could think of (caching, load balancing on cores, CPU frequency being altered due to coming out of sleep), but everything seems normal.

For what it's worth, I'm doing this testing on a school linux server, so it's possible there's an unusual configuration I don't know about, but the processor ID being used doesn't change, and (via top) the server was completely idle as I tested.

Test code:

#include <time.h>
#include <stdio.h>

#define MY_CLOCK CLOCK_MONOTONIC_RAW
// no difference if set to CLOCK_THREAD_CPUTIME_ID

typedef struct {
        unsigned int tsc;
        unsigned int proc;
} ans_t;

static ans_t rdtscp(void){
        ans_t ans;
        __asm__ __volatile__ ("rdtscp" : "=a"(ans.tsc), "=c"(ans.proc) : : "edx");
        return ans;
}

static void nop(void){
        __asm__ __volatile__ ("");
}

void test(){
        for(int i=0; i<100000000; i++) nop();
}

int main(){
        int c=10;
        while(c-->0){
                struct timespec tstart,tend;
                ans_t start = rdtscp();
                clock_gettime(MY_CLOCK,&tstart);
                test();
                ans_t end = rdtscp();
                clock_gettime(MY_CLOCK,&tend);
                unsigned int tdiff = (tend.tv_sec-tstart.tv_sec)*1000000000+tend.tv_nsec-tstart.tv_nsec;
                unsigned int cdiff = end.tsc-start.tsc;
                printf("%u cycles and %u ns (%lf GHz) start proc %u end proc %u\n",cdiff,tdiff,(double)cdiff/tdiff,start.proc,end.proc);
        }
}

Output I see:

351038093 cycles and 125680883 ns (2.793091 GHz) start proc 14 end proc 14
350911246 cycles and 125639359 ns (2.793004 GHz) start proc 14 end proc 14
350959546 cycles and 125656776 ns (2.793001 GHz) start proc 14 end proc 14
351533280 cycles and 125862608 ns (2.792992 GHz) start proc 14 end proc 14
350903833 cycles and 125636787 ns (2.793002 GHz) start proc 14 end proc 14
350924336 cycles and 125644157 ns (2.793002 GHz) start proc 14 end proc 14
349827908 cycles and 125251782 ns (2.792997 GHz) start proc 14 end proc 14
175289886 cycles and 62760404 ns (2.793001 GHz) start proc 14 end proc 14
175283424 cycles and 62758093 ns (2.793001 GHz) start proc 14 end proc 14
175267026 cycles and 62752232 ns (2.793001 GHz) start proc 14 end proc 14

I get similar output (with it taking a different number of tests to double in efficiency) using different optimization levels (-O0 to -O3).

Could it perhaps have something to do with hyperthreading, where two logical cores in a physical core (the server is using Xeon X5560s which may have this effect) can somehow "merge" to form one twice-as-fast processor?

Answer 1

Some systems scale the processor speed depending on the system load. As you justly note, this is particularly annoying when benchmarking.

If your server is running Linux, please type

cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor 

If this outputs ondemand, powersave or userspace, then CPU frequency scaling is active, and you're going to find it very difficult to do benchmarks. If this says performance, then CPU frequency scaling is disabled.