Why GDB cannot find the source lines of the Linux kernel when debugging through QEMU?

Question

I've been trying to get the kernel to execute within QEMU using GDB to remote debug it but I'm not able to step through or set breakpoints. Here's the GDB session:

linux (master *) $ gdb vmlinux
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
[LICENSE TEXT]
The target architecture is assumed to be i386:x86-64
Reading symbols from vmlinux...done.
(gdb) target remote localhost:1234
localhost:1234: Connection timed out.
(gdb) target remote localhost:1234
Remote debugging using localhost:1234
0xffffffffa13507ee in ?? ()
(gdb) c
Continuing.
^C
Thread 1 received signal SIGINT, Interrupt.
0xffffffffa13507ee in ?? ()
(gdb) b rcu_process_callbacks
Breakpoint 1 at 0xffffffff81101800: file kernel/rcu/tree.c, line 3037.
(gdb) c
Continuing.
Warning:
Cannot insert breakpoint 1.
Cannot access memory at address 0xffffffff81101800

Command aborted.
(gdb) 

I've compiled the kernel with debug symbols as mentioned in the kernel documentation:

linux (master *) $ grep CONFIG_DEBUG .config | grep -v "^#" 
CONFIG_DEBUG_DEVRES=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_FS=y
CONFIG_DEBUG_KERNEL=y
CONFIG_DEBUG_STACK_USAGE=y
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_DEBUG_STACKOVERFLOW=y
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_BOOT_PARAMS=y

I also checked if the build kernel has debugging symbols.

linux (master *) $ nm --debug-syms vmlinux | grep "\.debug"
0000000000000000 N .debug_abbrev
0000000000000000 N .debug_aranges
0000000000000000 N .debug_frame
0000000000000000 N .debug_info
0000000000000000 N .debug_line
0000000000000000 N .debug_loc
0000000000000000 N .debug_ranges
0000000000000000 N .debug_str

I'm running QEMU with the following command-line:

linux (master *) $ qemu-system-x86_64 -smp 4 -cpu host \
-m 2048 -kernel arch/x86/boot/bzImage \
-initrd ../obj/initramfs-busybox-x86.cpio.gz -nographic \
-append "console=ttyS0" -enable-kvm \
-drive file=../disk.img,if=virtio,cache=none -s

What am I missing which could cause GDB to fail like this? I'm also not able to execute GDB extension commands from the kernel GDB scripts, so I'm guessing this has to do GDB not being aware that it is debugging a kernel? Do I need to enable KGDB for this to work?

I'm using the Linux 4.12-rc5 kernel.

Answer 1

It is KASLR (Kernel address space layout randomization).

Because the $pc, $rip of program does not start with 0xffffffff81xxxxxx, rather it is 0xffffffffa1xxxxxx

In gdb, use add-symbol-file to add kernel symbol file instead of file (so that you can specify the address of .text .data .bss), or disable kaslr (i.e. nokaslr in kernel boot args).

(I prefer the former, kaslr is too interesting to be disabled.)

Answer 2

qemu-system-x86_64 -append nokaslr

The nokasrl Linux command line parameter disables KASRL.

This has been needed since v4.12, when KASLR was turned on by default.

Here is a highly automated Buildroot example that uses it.

Early boot source code

The very first instructions that run are actually an unpackaging routine for a compressed kernel into memory. I have not been able to GDB those with source yet. See also:

• https://unix.stackexchange.com/questions/5518/what-is-the-difference-between-the-following-kernel-makefile-terms-vmlinux-vml/482978#482978
• What are the first operations that the Linux Kernel executes on boot?