How to abort RSA_generate_key?

Question

Consider the following invocation of RSA_generate_key():

RSA * rsa = RSA_generate_key(8192, RSA_F4, NULL, NULL);

Generating an 8,192-bit RSA key can take a long time (anywhere from a few seconds to a few minutes). Suppose the application containing the line of code above offers the user a button that will cancel the key generation.

How can I abort the computation and make the function return before the key has been generated? I remember the third argument to RSA_generate_key() is a callback function used for displaying progress - is there any way that callback could return a value that means "abort the operation and return"?

Running the function in another thread and then terminating the thread is not an option.

Answer 1

Since RSA_generate_key provides a progress callback, you can longjmp out of it to terminate the function. With a bit of additional code, you can create a wrapper for RSA_generate_key that accepts a generic test function which can be used to check for a timeout or a flag set by the windowing system.

#include <openssl/rsa.h>
#include <stdbool.h>
#include <setjmp.h>

struct trampoline_ctx {
  bool (*testfn)(void *);
  void *testfn_arg;
  jmp_buf env;
};

static void trampoline(int ignore1, int ignore2, void *arg)
{
  struct trampoline_ctx *ctx = arg;
  if (!ctx->testfn(ctx->testfn_arg))
    longjmp(ctx->env, 1);
}

// like RSA_generate_key, but accepts a test function. If testfn returns
// false, key generation is terminated and NULL is returned.    
RSA *
my_generate_key(int num, unsigned long e,
                bool (*testfn)(void *), void *testfn_arg)
{
  struct trampoline_ctx ctx;
  ctx.testfn = testfn;
  ctx.testfn_arg = testfn_arg;
  if (setjmp(ctx.env))
    return NULL;
  return RSA_generate_key(num, e, trampoline, &ctx);
}

This approach is surprisingly portable, as longjmp is mandated by both C89 and C99. Its disadvantage is that it can leak resources if the function you're longjmping out of allocates them dynamically. However, in practice the leak can be small enough to be unnoticable if done infrequently or only on explicit user request. To be positive that this is the case, run the code in a tight loop and observe resource consumption of the process.

Here is a test program for the above function:

#include <stdio.h>
#include <sys/time.h>

double now()
{
  struct timeval tv;
  gettimeofday(&tv, NULL);
  return tv.tv_sec + (double) tv.tv_usec / 1e6;
}

struct tt_ctx {
  double start;
  double limit;
};

bool test_time_limit(void *arg)
{
  struct tt_ctx *ctx = arg;
  return now() - ctx->start <= ctx->limit;
}

int main(int argc, char **argv)
{
  int limit = atoi(argv[1]);
  struct tt_ctx ctx;
  ctx.start = now();
  ctx.limit = limit / 1000.0;

  RSA *key = my_generate_key(4096, 65537, test_time_limit, &ctx);
  printf("%p\n", key);
  return 0;
}

The test program assumes POSIX gettimeofday, but can be trivially converted to another high-resolution clock as provided by the system. Test procedure is as follows:

Append both pieces of code to a file and compile with -lrsa. The test program will generate a 4096-bit RSA key within the time limit specified in milliseconds on the command line. In all cases it prints the resulting RSA * pointer to indicate whether my_generate_key completed its request or it was aborted. The output of time accompanies execution as a sanity check to verify that the time limit is being respected:

# try with a 10ms limit
$ time ./a.out 10
(nil)                         # too short
./a.out 10  0.02s user 0.00s system 85% cpu 0.023 total

# see if 100ms is enough time
$ time ./a.out 100
(nil)                         # still too short
./a.out 100  0.10s user 0.00s system 97% cpu 0.106 total

# try with 1 whole second:
$ time ./a.out 1000
0x2369010                     # success!
./a.out 1000  0.64s user 0.00s system 99% cpu 0.649 total

Answer 2

The progress callback can't be used to cancel the function, it's purely for displaying the progress to the user. (Read more here)

Spawn a thread to call RSA_generate_key() so that it executes in the background. When the user presses the cancel button, kill the thread.

Answer 3

I believe the problem is that a callback doesn't generally give you the ability to send information in, only receive information back. As some of the other posters have mentioned, you might consider altering the source code of the actual key generation routine, and compiling your own version. In this case, you could pass some kind of a value by reference back to your calling routine, which you could then set outside, and thus trigger a fail condition.

Note: RSA_generate_key is actually deprecated, and is now simply a wrapper for RSA_generate_key_ex.

According to version 1.19.4.2 of the rsa_gen.c file, unless you are in FIPS_mode, the key is going to be generated by the static method rsa_builtin_keygen. In FIPS_mode, it will be generated by whatever rsa->meth->rsa_keygen is upon entering the RSA_generate_key.

That said, there are a number of places where the callback (cb parameter) is passed out to other methods, presumably so that those methods can, in turn, call it to update status.

The trick would be updating either the BN_GENCB to contain some kind of a "cancel" flag, or alter the way in which the callback is actually called within the method, so that when the callback method fires, you can set a flag, which the calling function would honor by breaking out of the generation subroutine.

How you'd go about that is still something you need to figure out and work through - but hopefully this will give you a bit of a start.

Answer 4

Although I never face such situation and never did anything before. You can try the following if you can modify the source code of OpenSSL:

Let us say a variable V is shared between two functions. So either it is a global variable or static variable in the file and being used by RSA_generate_key and say another function RSA_set_V.

RSA_generate_key will only read the variable. RSA_set_V sets the variable.

In RSA_generate_key, the time consuming loop may use an additional condition of checking this variable. It may terminate this loop or exit the function on a certain value of V. e.g.

  while (V!=CERTAIN_VALUE && Rest_of_Condition) {
      //Loop body.
  }

Now, in RSA_set_V, set V to CERTAIN_VALUE. When user clicks on Stop, call RSA_set_V. It should stop RSA_generate_key.

However, it has few problems. It may slow down RSA_generate_key.