kvm hypercall with multiple args

Question

I'm currently trying to build a small hypervisor and kernel using kvm and I struggle to get hypercalls with multiple args working correctly.

Here is what I've tried:

// guest.c

#define KVM_HYPERCALL vmcall
// #define KVM_HYPERCALL vmmcall
// #define KVM_HYPERCALL ".byte 0x0f,0x01,0xd9"
// #define KVM_HYPERCALL .byte 0x0f,0x01,0xc1"

static inline long kvm_hypercall4(int nr, unsigned long p1,
                  unsigned long p2, unsigned long p3,
                  unsigned long p4) {
    long ret;
    asm volatile(KVM_HYPERCALL
             : "=a"(ret)
             : "a"(nr), "b"(p1), "c"(p2), "d"(p3), "S"(p4)
             : "memory");
    return ret;
}

Any of those Hypercalls lead to vcpu->kvm_run->exit_reason to equal 6 which is to my surprise KVM_EXIT_MMIO instead of KVM_EXIT_HYPERCALL

switch (vcpu->kvm_run->exit_reason) {
  case KVM_EXIT_MMIO:
    printf("syscall: %lld\n", vcpu->kvm_run->hypercall.nr); // prints 0
    printf("arg 1: %lld\n",  vcpu->kvm_run->hypercall.args[1]); // prints 0
    printf("arg 2: %lld\n", vcpu->kvm_run->hypercall.args[2]); // prints 0
    printf("arg 3: %lld\n",  vcpu->kvm_run->hypercall.args[3]); // prints 0

    if(ioctl(vcpu->fd, KVM_GET_REGS, &regs)<0) exit 1;

    printf("rax: %lld\n", regs.rax); // prints 0
    printf("rbx: %lld\n", regs.rbx); // prints 0
    printf("rcx: %lld\n", regs.rcx); // prints 0

Aside of the exit reason being KVM_EXIT_MMIO why are the regs not set? What is the right way to trigger a KVM_EXIT_HYPERCALL with multple arguments?

Thanks in advance

EDIT: In case it matters: I'm using 9th generation intel i7 cpu running debian with linux kernel 5.4

Answer 1

KVM_EXIT_HYPERCALL is no longer used, according to the documentation:

/* KVM_EXIT_HYPERCALL */
      struct {
          __u64 nr;
          __u64 args[6];
          __u64 ret;
          __u32 longmode;
          __u32 pad;
      } hypercall;

Unused. This was once used for 'hypercall to userspace'. To implement such functionality, use KVM_EXIT_IO (x86) or KVM_EXIT_MMIO (all except s390). Note KVM_EXIT_IO is significantly faster than KVM_EXIT_MMIO.

And it seems to me that KVM_EXIT_HYPERCALL is also not implemented. Quick and dirty search with grep. It's defined, but will never assigned as exit_reason:

user@host:~/Linux/src> grep -R KVM_EXIT_HYPERCALL
include/uapi/linux/kvm.h:#define KVM_EXIT_HYPERCALL        3
include/uapi/linux/kvm.h:               /* KVM_EXIT_HYPERCALL */
Documentation/virt/kvm/api.rst:         /* KVM_EXIT_HYPERCALL */
tools/include/uapi/linux/kvm.h:#define KVM_EXIT_HYPERCALL        3
tools/include/uapi/linux/kvm.h:         /* KVM_EXIT_HYPERCALL */
tools/testing/selftests/kvm/lib/kvm_util.c:     {KVM_EXIT_HYPERCALL, "HYPERCALL"},
user@host:~/Linux/src>

Linux version:

user@host:~/Linux/src> git-describe --tags
v5.6-10895-g4c205c84e249
user@host:~/Linux/src>

There is an older question how to implement custom VMCALL's with two answers on this site. Have you tried them?