QEMU - Code Flow [ Instruction cache and TCG]

Question

I am trying to analyze the QEMU source code.

I know its huge and till date there is no official documentation for it.

My main areas of concern are the Instruction cache management and TCG operation.

Any pointers to them would be helpful ?

Answer 1

I know full answer would be much longer, but for start I just want to bring to your attention this diagram: (now, it would be useful for you to play with gdb running QEMU, set breakpoints in functions you see in the diagram, follow code execution, etc.)

Answer 2

Yes, the QEMU code flow has changed a good bit. I'm not going to do a libreoffice presentation but here's a couple TCG stack traces of the QEMU 5.1 MTTCG code. The first is the TCG Front-end (FE) taking the guest code and converting it to internal intermediate code in a Translation Block (TB). The TB has a max size of 512 instructions.

#0  0x00005555559ce81f in disas_insn (s=0x7fffe933d450, cpu=0x555556b2d990) at /opt/distros/qemu-5.1.0/target/i386/translate.c:4476
#1  0x00005555559dd471 in i386_tr_translate_insn (dcbase=0x7fffe933d450, cpu=0x555556b2d990) at /opt/distros/qemu-5.1.0/target/i386/translate.c:8569
#2  0x00005555558c4222 in translator_loop (ops=0x5555565ae9a0 <i386_tr_ops>, db=0x7fffe933d450, cpu=0x555556b2d990, tb=0x7fffac099900 <code_gen_buffer+134846675>, max_insns=512) at /opt/distros/qemu-5.1.0/accel/tcg/translator.c:102
#3  0x00005555559dd643 in gen_intermediate_code (cpu=0x555556b2d990, tb=0x7fffac099900 <code_gen_buffer+134846675>, max_insns=512) at /opt/distros/qemu-5.1.0/target/i386/translate.c:8631
#4  0x00005555558c2258 in tb_gen_code (cpu=0x555556b2d990, pc=18446744071591428680, cs_base=0, flags=4244144, cflags=-16252928) at /opt/distros/qemu-5.1.0/accel/tcg/translate-all.c:1743
#5  0x00005555558be77a in tb_find (cpu=0x555556b2d990, last_tb=0x0, tb_exit=0, cf_mask=524288) at /opt/distros/qemu-5.1.0/accel/tcg/cpu-exec.c:407
#6  0x00005555558bf18e in cpu_exec (cpu=0x555556b2d990) at /opt/distros/qemu-5.1.0/accel/tcg/cpu-exec.c:748
#7  0x00005555559846eb in tcg_cpu_exec (cpu=0x555556b2d990) at /opt/distros/qemu-5.1.0/softmmu/cpus.c:1356
#8  0x0000555555984f41 in qemu_tcg_cpu_thread_fn (arg=0x555556b2d990) at /opt/distros/qemu-5.1.0/softmmu/cpus.c:1664
#9  0x0000555555e5ec8d in qemu_thread_start (args=0x555556b5e0e0) at /opt/distros/qemu-5.1.0/util/qemu-thread-posix.c:521
#10 0x00007ffff3c406db in start_thread (arg=0x7fffe933e700) at pthread_create.c:463
#11 0x00007ffff396971f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

The second is the TCG Back-end (BE) translating the intermediate code into host instructions.

Thread 3 "qemu-system-x86" hit Breakpoint 9, tcg_out_op (s=0x7ffe9c000b20, opc=INDEX_op_ld_i32, args=0x7fffe933d530, const_args=0x7fffe933d4f0) at /opt/distros/qemu-5.1.0/tcg/i386/tcg-target.inc.c:2259
2259        int c, const_a2, vexop, rexw = 0;
#0  0x00005555558447d2 in tcg_out_op (s=0x7ffe9c000b20, opc=INDEX_op_ld_i32, args=0x7fffe933d530, const_args=0x7fffe933d4f0) at /opt/distros/qemu-5.1.0/tcg/i386/tcg-target.inc.c:2259
#1  0x000055555584fb40 in tcg_reg_alloc_op (s=0x7ffe9c000b20, op=0x7ffe9c00a418) at /opt/distros/qemu-5.1.0/tcg/tcg.c:3803
#2  0x000055555585078e in tcg_gen_code (s=0x7ffe9c000b20, tb=0x7fffac129880 <code_gen_buffer+135436371>) at /opt/distros/qemu-5.1.0/tcg/tcg.c:4244
#3  0x00005555558c22f1 in tb_gen_code (cpu=0x555556b2d990, pc=94290869746347, cs_base=0, flags=4244147, cflags=-16252928) at /opt/distros/qemu-5.1.0/accel/tcg/translate-all.c:1766
#4  0x00005555558be77a in tb_find (cpu=0x555556b2d990, last_tb=0x7fffac0ea700 <code_gen_buffer+135177939>, tb_exit=1, cf_mask=524288) at /opt/distros/qemu-5.1.0/accel/tcg/cpu-exec.c:407
#5  0x00005555558bf18e in cpu_exec (cpu=0x555556b2d990) at /opt/distros/qemu-5.1.0/accel/tcg/cpu-exec.c:748
#6  0x00005555559846eb in tcg_cpu_exec (cpu=0x555556b2d990) at /opt/distros/qemu-5.1.0/softmmu/cpus.c:1356
#7  0x0000555555984f41 in qemu_tcg_cpu_thread_fn (arg=0x555556b2d990) at /opt/distros/qemu-5.1.0/softmmu/cpus.c:1664
#8  0x0000555555e5ec8d in qemu_thread_start (args=0x555556b5e0e0) at /opt/distros/qemu-5.1.0/util/qemu-thread-posix.c:521
#9  0x00007ffff3c406db in start_thread (arg=0x7fffe933e700) at pthread_create.c:463
#10 0x00007ffff396971f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Once the TB is generated then tb_find returns with it and cpu_tb_exec runs it.

As prior answers indicate, there is A LOT more to the TCG.

Note that I used host and guest rather than native and target. TCG has the terms somewhat reversed: the source is the guest code that is converted to run on the target, which is the host. In other words, the TCG target is the QEMU host (which makes sense for the VM code generation.)