Omitting core dumps via C a #define (or other in code/compile time solutions)?

Question

I understand that the operating system generates a core dump sometimes when a signal is sent (usually upon a crash). Is there a way to tell the operating system from inside C/C++ via a #define or compiler flag that no, or a somehow specifically limited core dump can be generated via that executable? The only way I know of to control it is via ulimit -c. My test system is Linux. The no core dump does not need to be implemented system wide, only for a specific program.

For the interested, this has to do with CVE-2019-15947 in Bitcoin Core's bitcoin-qt which still has no solution.

Some of this discussion is at the Bitcoin GitHub bug tracking page.

The other option would be to obfunscate and or encrypt the wallet.dat in memory so it's not easily retrievable via core dumps. Please note the second option can already be accomplished, though is not enabled by default.

Answer 1

Depending on your definition of "in code/compile-time", you can install a signal handler and wipe memory upon receiving that signal.

However, crashes are handled by the kernel, not the compiler or the executable. You cannot stop memory from being dumped by the kernel into a core from inside the executable, no matter what you do.

Therefore, the other option sounds best.

Answer 2

The key primitive you will want to use is madvise(..., MADV_DONTDUMP), which notifies Linux (since 3.4) that you do not wish a series of pages to be included in the dump. The flag is also called VM_DONTDUMP in kernel space. (Note that some versions of gdb do not respect this flag, which could be relevant for cores generated by gcore or other helpers rather than by the kernel.)

You will also need to ensure that when processing keys and other sensitive data stored in these pages information is not disclosed via registers or spilled to the stack sufficient for a compromise should a core dump after that time.