I am trying to use trace-cmd
to gather more information about a kernel crash that I am seeing. Unfortunately, kernel crashes with "kernel panic - not syncing" message (i.e. socket and file buffers are not flushed so whatever was in the buffers at the time of crash is inevitably lost).
Is there a way to:
trace-cmd
process to flush its buffers to filesystem or sockets? ORSince I am running that trace-cmd
command in virtualized environment then I don't care that hard drive may occasionally get corrupted, because I can simply revert to last good snapshot or redeploy the VM.
Ftrace uses a combination of both static probes (function tracing, event tracing, etc) and dynamic probes (kprobes, uprobes, etc). To trace function calls, ftrace will build the kernel with GCC’s -pg option.
See Function Tracer Design for details for arch porters and such. Ftrace uses the tracefs file system to hold the control files as well as the files to display output. When tracefs is configured into the kernel (which selecting any ftrace option will do) the directory /sys/kernel/tracing will be created.
trace-cmd is an easy-to-use, feature-rich utility for tracing Linux kernel functions. Get the highlights in your inbox every week. In my previous article, I explained how to use ftrace to trace kernel functions.
As ftrace provides a function tracer, it makes it convenient to check the stack size at every function call. This is enabled via the stack tracer. CONFIG_STACK_TRACER enables the ftrace stack tracing functionality.
I can't guarantee my recipe helps you, but who knows :)
At first try to unbuffer all output of trace-cmd
. You could find some recipes to do this there https://unix.stackexchange.com/questions/25372/turn-off-buffering-in-pipe
Second, kernel treats writing FS on panic is very dangerous and that's ok, however you could try to store last bites of info before death outside of crashed machine. Try to redirect your critical info to serial-port or network connection and catch it on the other side. Like
unbuffer trace-cmd >/dev/ttyS0
Another way is to try avoid panic at all and let kernel works longer however with unpredictable result. You can disable panic for several cases. Look at panic_on_*
settings in /proc/sys/kernel/
directory. More details can be found in the "Documentation for /proc/sys/kernel/"
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With