mbind returns EINVAL

Question

I am using the code provided for the following question numa+mbind+segfault, every call to mbind returns EINVAL. How can I get what is exactly wrong? I am asking this because EINVAL can be returned for many reasons.

page_size = sysconf(_SC_PAGESIZE);
objs_per_page = page_size/sizeof(A[0]);
assert(page_size%sizeof(A[0])==0);
split_three=num_items/3;
aligned_size=(split_three/objs_per_page)*objs_per_page;
remnant=num_items-(aligned_size*3);
piece = aligned_size;

nodemask=1;
mbind(&A[0],piece*sizeof(double),MPOL_BIND,&nodemask,64,MPOL_MF_MOVE);

nodemask=2;
mbind(&A[aligned_size],piece*sizeof(double),MPOL_BIND,&nodemask,64,MPOL_MF_MOVE);

nodemask=4;
bind(&A[aligned_size*2+remnant],piece*sizeof(double),MPOL_BIND,
     &nodemask,64,MPOL_MF_MOVE);

After running the program (by changing the nodemask before every mbind call to 1,2 and 4 respectively) shown below (as an answer from Mats Petersson). It sometimes segfaults and sometimes runs fine. When it segfaults the dmesg is as follows:

Stack:
Call Trace:
mpol_new+0x5d/0xb0
sys_mbind+0x125/0x4f0
finish_task_switch+0x4a/0xf0
? __schedule+0x3cf/0x7c0
system_call_fastpath+0x16/0x1b
Code: ...
kmem_cache_alloc+0x58/0x130

Answer 1

Looking at the source of Linux kernel, you can get EINVAL for:

• Passing in an invalid mode value. Either out of range of "inconsistent" (using both static and relative nodes at the same time)
• invalid maxnode (> number of bits in a page -> 32K on x86).
• Various other problems with nodemask.
• Not having one of MPOL_MF_STRICT | MPOL_MF_MOVE | MPOL_MF_MOVE_ALL
• start is not page-aligned.
• start+len when page-aligned = start. [that is, your len is not at least one byte]
• start+len < start - that is, negative length.
• policy = MPOL_DEFAULT and nodes isn't empty or NULL.
• quoting comment from source "MPOL_PREFERRED cannot be used with MPOL_F_STATIC_NODES or MPOL_F_RELATIVE_NODES if the nodemask is empty (local allocation). All other modes require a valid pointer to a non-empty nodemask.

My guess would be on start is not page-aligned.

This code works for me:

#include <numaif.h>
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>

#define ASSERT(x) do { if (!(x)) do_assert(#x,(long)(x), __FILE__, __LINE__); } while(0)

static void do_assert(const char *expr, long expr_int, const char *file, int line)
{
    fprintf(stderr, "ASSERT failed %s (%d) at %s:%d\n", 
        expr, expr_int, file, line);
    perror("Error if present:");
    exit(1);
}


int main()
{ 
    size_t num_items = 6156000;
    double *A = valloc(num_items * sizeof(double));
    ASSERT(A != NULL);
    int res;
    unsigned long nodemask;


    size_t page_size = sysconf(_SC_PAGESIZE);
    size_t objs_per_page = page_size/sizeof(A[0]);
    ASSERT(page_size%sizeof(A[0])==0);
    size_t split_three=num_items/3;
    size_t aligned_size=(split_three/objs_per_page)*objs_per_page;
    size_t remnant=num_items-(aligned_size*3);
    size_t piece = aligned_size;

    printf("A[0]=%p\n", &A[0]);
    printf("A[%d]=%p\n", piece, &A[aligned_size]);
    printf("A[%d]=%p\n", 2*piece, &A[2*piece]);


    nodemask=1;
    res = mbind(&A[0],piece*sizeof(double),MPOL_BIND,&nodemask,64,MPOL_MF_MOVE);
    ASSERT(res ==0);
    nodemask=1;
    res = mbind(&A[aligned_size],piece*sizeof(double),MPOL_BIND,&nodemask,64,MPOL_MF_MOVE);
    ASSERT(res ==0);

    nodemask=1;
    res = mbind(&A[aligned_size*2],(piece+remnant)*sizeof(double),MPOL_BIND,
     &nodemask,64,MPOL_MF_MOVE);
    ASSERT(res == 0);
}

Note that I'm using "nodemask=1" on all allocations, since I've only got one quad-core processor in my machine, so no other nodes to bind to - which also gives EINVAL. I take it you actually have more than one node in your system.

I also moved the "remnant" from the A[] to piece+remnant size for the last mbind call.