How to set conditional breakpoint if malloc returns NULL via gdb

Question

Sample source code:

#include        <stdio.h>
#include        <stdlib.h>
#include        <errno.h>

#define         GIGABYTE        1024*1024*1024

int
main (void)
{
        void    *foo;
        int     result;

        foo = (void *) malloc (GIGABYTE*5);
        result = errno;

        if (foo != NULL) {
                return 2;
        } else {
                fprintf (stderr, "ERROR: %d\n", result);
                return 1;
        }
        return 0;
}

Question:

• How to instruct gdb (# gdb -silent ./huge_malloc) to stop/halt execution, if malloc() returns 0x0, without checking if foo is 0x0

Answer 1

You could identify the exit point of malloc and put a conditional breakpoint there. Such as:

(gdb) tbreak main
Breakpoint 1 at 0x4005c4: file t.c, line 13.
(gdb) r
Starting program: /var/tmp/a.out 
main () at t.c:13
13          foo = malloc (64);
(gdb) br *__libc_malloc+211 if $rax==0
Breakpoint 2 at 0x7f26d143ea93
(gdb) n
14          foo = malloc (GIGABYTE*64);
(gdb) p foo
$1 = (void *) 0x21dc010
(gdb) n

Breakpoint 2, 0x00007f26d143ea93 in malloc () from /lib/libc.so.6

Note, I have added a malloc call that succeeds first, to illustrate that the breakpoint only triggers for a NULL return value. The breakpoint address may vary with libc versions, I found it by stepping through malloc with nexti until I hit the ret instruction.