Asynchronous evaluation in Mathematica

Question

Sometimes when I'm writing experimental code in Mathematica, I'm wary as to whether I should evaluate it or not because it may end up bringing my system to it's knees.

As a contrived example if you try running the following snippet of code on a 64-bit machine, it will most likely cause your system to grind to a complete halt after it eats up all your memory.

junk = Table[{x, x}, {10^9}]; (* nom nom nom memory. Please don't run this. *)

Sure, you can just throw MemoryConstrained onto it and hope for the best, but sometimes you don't want it blocking any further input. To that end, the way I thought it might be best to achieve a middle ground was to perform the evaluation in a separate kernel.

That was decently easy enough to do:

ClearAll[GetAvailableKernel];
GetAvailableKernel[] := Block[{i, kernels},
  kernels = Kernels[];
  If[Length@kernels !=  0,
   For[i = 1, i <= Length@kernels, i++,
    If[kernels[[i, 1, 2]] > 0, Return@kernels[[i]]]
    ]
   ];
  LaunchKernels[1]]

ClearAll[SafeEvaluate];
SetAttributes[SafeEvaluate, HoldFirst];
Options[SafeEvaluate] = {"EvaluationKernel" -> Null, 
   "ConstrainMemory" -> True, "MaxMemory" -> 2 1024^3};

SafeEvaluate[expr_, OptionsPattern[]] := Block[{evalkernel, result},
  If[OptionValue["EvaluationKernel"] != Null,
   evalkernel = OptionValue["EvaluationKernel"],
   evalkernel = GetAvailableKernel[]
   ];

  result  = If[OptionValue["ConstrainMemory"],
    With[{memory = OptionValue["MaxMemory"]},
     ParallelEvaluate[MemoryConstrained[expr, memory], evalkernel]],
    ParallelEvaluate[expr, evalkernel]];
  result]

Then you could just go ahead and do something along the lines of:

SafeEvaluate[Table[{x, x}, {1024^3}]]

And Mathematica would gracefully return $Aborted telling you it ran out of memory. By evaluating in a separate kernel, we can sandbox code into it's own parallel kernel. If something goes wrong, then our main kernel isn't affected.

---

This brings me to my main point: How can I achieve asynchronous evaluation within Mathematica?

What I have right now works, but it completely blocks any further user input. I can't just set, forget, and check later.

Any thoughts?

Answer 1

I have next to zero experience with parallel computation in Mathematica, so this might not be the best way, but this is what I managed to dig up from the docs:

Launch a kernel:

In[1]:= LaunchKernels[1]

Out[1]= KernelObject[1, "local"]

Submit some long to finish job:

In[2]:= job = 
 ParallelSubmit[First@SingularValueList[RandomReal[1, {2000, 2000}]]]

Start job:

In[3]:= Parallel`Developer`QueueRun[]

Out[3]= True

Now the job is running in parallel in the background ...

... and we're free to do whatever we want in the main kernel. If I understand your question, this is what you needed. We can run Parallel`Developer`QueueRun[] again to check which parallel evaluations have finished (the display of the evaluation object will dynamically update).

In[4]:= 1 + 1

Out[4]= 2

Wait until the evaluation finishes (if it hasn't yet) and collect the result:

In[5]:= WaitAll[job]

Out[5]= 1000.23