Confusing result from counting page fault in linux

Question

I was writing programs to count the time of page faults in a linux system. More precisely, the time kernel execute the function __do_page_fault.
And somehow I wrote two global variables, named pfcount_at_beg and pfcount_at_end, which increase once when the function __do_page_fault is executed at different locations of the function.

To illustrate, the modified function goes as:

unsigned long pfcount_at_beg = 0;
unsigned long pfcount_at_end = 0;
static void __kprobes
__do_page_fault(...)
{
    struct vm_area_sruct *vma;
    ...    // VARIABLES DEFINITION 
    unsigned int flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;   
    pfcount_at_beg++;        // I add THIS
    ...
    ... 
    // ORIGINAL CODE OF THE FUNCTION
    ...
    pfcount_at_end++;        // I add THIS
}

I expected that the value of pfcount_at_end is smaller than the value of pfcount_at_beg.

Because, I think, every time kernel executes the instructions of code pfcount_at_end++, it must have executed pfcount_at_beg++(Every function starts at the very beginning of the code).
On the other hand, as there are many conditional return between these two lines of code.

However, the result turns out oppositely. The value of pfcount_at_end is larger than the value of pfcount_at_beg.
I use printk to print these kernel variables through a self-defined syscall. And I wrote the user level program to call the system call.

Here is my simple syscall and user-level program:

// syscall
asmlinkage int sys_mysyscall(void)
{
    printk( KERN_INFO "total pf_at_beg%lu\ntotal pf_at_end%lu\n", pfcount_at_beg, pfcount_at_end)
    return 0;
}

// user-level program
#include<linux/unistd.h>
#include<sys/syscall.h>
#define __NR_mysyscall 223
int main()
{
    syscall(__NR_mysyscall);
    return 0;
}

Is there anybody who knows what exactly happened during this?

Just now I modified the code, to make pfcount_at_beg and pfcount_at_end static. However the result did not change, i.e. the value of pfcount_at_end is larger than the value of pfcount_at_beg. So possibly it might be caused by in-atomic operation of increment. Would it be better if I use read-write lock?

Answer 1

The ++ operator is not garanteed to be atomic, so your counters may suffer concurrent access and have incorrect values. You should protect your increment as a critical section, or use the atomic_t type defined in <asm/atomic.h>, and its related atomic_set() and atomic_add() functions (and a lot more).

Not directly connected to your issue, but using a specific syscall is overkill (but maybe it is an exercise). A lighter solution could be to use a /proc entry (also an interesting exercise).