eBPF: BPF stack limit exceeded when storing stack variable in map

Question

I have the following eBPF program:

#include <stdio.h>
#include <string.h>

#include <linux/bpf.h>
#include <sys/socket.h>

#include <bpf/bpf_helpers.h>

char LICENSE[] SEC("license") = "GPL";

// msg_data_map carries a key-value pair of (msg_id, msg_length), and can record
// upto 65535 messages at once.
#define MAX_MSG_LEN 128
struct {
  __uint(type, BPF_MAP_TYPE_HASH);
  __uint(max_entries, 65535);
  __type(key, int);
  __type(value, char[MAX_MSG_LEN]);
} msg_data_map SEC(".maps");

SEC("sk_msg")
int msg_prog(struct sk_msg_md *msg) {
  long len = (long)msg->data_end - (long)msg->data;

  void *data_end = (void *)(long) msg->data_end;
  void *data = (void *)(long) msg->data;

  // Bounds check to make verifier happy
  if (data + MAX_MSG_LEN > data_end) {
    return SK_PASS;
  }

  char buf[MAX_MSG_LEN] = {0};
  if (len > MAX_MSG_LEN) {
    __builtin_memcpy(buf, data, MAX_MSG_LEN);
  } else {
    __builtin_memcpy(buf, data, len);
  }

  // Update in map
  int index = 0;
  bpf_map_update_elem(&msg_data_map, &index, &buf, BPF_ANY);
 
  return SK_PASS;
}

Compiling the above program gives the following error:

Looks like the BPF stack limit of 512 bytes is exceeded. Please move large on stack variables into BPF per-cpu array map.

1. Since the buf array is only 128 bytes, it should not have exceeded the stack limit.
2. If I comment the map update lines, the program compiles fine. Why is this the case?

Answer 1

The error message

Looks like the BPF stack limit of 512 bytes is exceeded. Please move large on stack variables into BPF per-cpu array map.

Is documented in bpftrace as well https://github.com/iovisor/bpftrace/blob/master/docs/internals_development.md#stack-limit-exceeded

It seems to be the nicer version of the error normally thrown by the kernel

combined stack size of %d calls is %d. Too large

but I am unable to find where this nicer version actually originates. In any case, its a valid error. The verifier enforces a 512 byte stack limit per stack frame (per BPF-to-BPF function).

The only way to get rid of this error is to decrease the amount of variables on the stack. The error suggests to use a per-CPU map to store some of the data currently on the stack since BPF programs are not preempted and thus the map value can be used without fear of conflicting with other program.

I should note that on RT (Real Time) kernels this assumption about per-CPU maps isn't true since BPF programs are never migrated but can still be preempted.