Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

use trace-cmd/ftrace to get function_graph just before panic() happened

I am trying to use trace-cmd to gather more information about a kernel crash that I am seeing. Unfortunately, kernel crashes with "kernel panic - not syncing" message (i.e. socket and file buffers are not flushed so whatever was in the buffers at the time of crash is inevitably lost).

Is there a way to:

  1. force trace-cmd process to flush its buffers to filesystem or sockets? OR
  2. make trace-cmd/ftrace to avoid buffering (ie call printk() behind scenes and use netconsole)?

Since I am running that trace-cmd command in virtualized environment then I don't care that hard drive may occasionally get corrupted, because I can simply revert to last good snapshot or redeploy the VM.

like image 825
user389238 Avatar asked Sep 26 '17 18:09

user389238


People also ask

How do I trace a function call in ftrace?

Ftrace uses a combination of both static probes (function tracing, event tracing, etc) and dynamic probes (kprobes, uprobes, etc). To trace function calls, ftrace will build the kernel with GCC’s -pg option.

How does ftrace work in Linux?

See Function Tracer Design for details for arch porters and such. Ftrace uses the tracefs file system to hold the control files as well as the files to display output. When tracefs is configured into the kernel (which selecting any ftrace option will do) the directory /sys/kernel/tracing will be created.

What is tracetrace-CMD and how to use it?

trace-cmd is an easy-to-use, feature-rich utility for tracing Linux kernel functions. Get the highlights in your inbox every week. In my previous article, I explained how to use ftrace to trace kernel functions.

What is stack tracer in ftrace?

As ftrace provides a function tracer, it makes it convenient to check the stack size at every function call. This is enabled via the stack tracer. CONFIG_STACK_TRACER enables the ftrace stack tracing functionality.


1 Answers

I can't guarantee my recipe helps you, but who knows :)

At first try to unbuffer all output of trace-cmd. You could find some recipes to do this there https://unix.stackexchange.com/questions/25372/turn-off-buffering-in-pipe

Second, kernel treats writing FS on panic is very dangerous and that's ok, however you could try to store last bites of info before death outside of crashed machine. Try to redirect your critical info to serial-port or network connection and catch it on the other side. Like

unbuffer trace-cmd >/dev/ttyS0

Another way is to try avoid panic at all and let kernel works longer however with unpredictable result. You can disable panic for several cases. Look at panic_on_* settings in /proc/sys/kernel/ directory. More details can be found in the "Documentation for /proc/sys/kernel/"

like image 97
MrCryo Avatar answered Sep 17 '22 13:09

MrCryo