avoiding abort in libgmp

Question

I have some code that uses libgmp. At some point the user may request a factorial of a very large number. Unfortunately, this results in libgmp raising an abort signal.

For example the following code:

#include <cmath>
#include <gmp.h>
#include <iostream>

int main() {

    mpz_t result;
    mpz_init(result);

    mpz_fac_ui(result, 20922789888000);

    std::cout << mpz_get_si(result) << std::endl;
}

Results in:

$ ./test 
gmp: overflow in mpz type
Aborted

Apparently, the number produced is REALLY big. Is there anyway to handle the error more gracefully than an abort. This is a GUI based application and it aborting is pretty much the least desirable way to handle this sort of issue.

Answer 1

It would appear that you are out of luck, based on the code in mpz/realloc.c and mpz/realloc2.c. If too much memory was requested, it just does this:

if (UNLIKELY (new_alloc > INT_MAX))
  {
    fprintf (stderr, "gmp: overflow in mpz type\n");
    abort ();
  }

Answer 2

The best way to handle these errors gracefully in your application is probably to fork off a helper process to perform the GMP calculations. If the helper process is killed by SIGABRT, your parent process can detect that and report an error to the user.

---

(The below is my original answer, which has "undefined results" according to the GMP documentation - it is left here for completeness).

You can catch the error if you install a signal handler for SIGABRT that uses longjmp():

jmp_buf abort_jb;

void abort_handler(int x)
{
    longjmp(abort_jb, 1);
}

int dofac(unsigned long n)
{
    signal(SIGABRT, abort_handler);
    if (setjmp(abort_jb))
        goto error;

    mpz_t result;
    mpz_init(result);

    mpz_fac_ui(result, 20922789888000);

    std::cout << mpz_get_si(result) << std::endl;

    signal(SIGABRT, SIG_DFL);
    return 0;

    error:
    signal(SIGABRT, SIG_DFL);
    std::cerr << "Caught SIGABRT from GMP.\n";
    return 1;
}

Answer 3

Overwrite abort() with LD_PRELOAD.

What is the LD_PRELOAD trick?

Edit: To make the answer more self-contained, I copy the text of that answer here:

If you set LD_PRELOAD to the path of a shared object, that file will be loaded before any other library (including the C runtime, libc.so). So to run ls with a your special malloc() implementation, do this:

$ LD_PRELOAD=/path/to/my/malloc.so /bin/ls

Credits to JesperE.