Monitoring progress of a parallel computation in Mathematica

Question

I'm building a large ParallelTable, and would like to maintain some sense of how the computation is going. For a non parallel table the following code does a great job:

counter = 1;
Timing[
 Monitor[
  Table[
   counter++
  , {n, 10^6}];
 , ProgressIndicator[counter, {0, 10^6}]
 ]
]

with the result {0.943512, Null}. For the parallel case, however, it's necessary to make the counter shared between the kernels:

counter = 1;
SetSharedVariable[counter];
Timing[
 Monitor[
  ParallelTable[
   counter++
  , {n, 10^4}];
 , ProgressIndicator[counter, {0, 10^4}]
 ]
]

with the result {6.33388, Null}. Since the value of counter needs to be passed back and forth between the kernels at every update, the performance hit is beyond severe. Any ideas for how to get some sense of how the computation is going? Perhaps letting each kernel have its own value for counter and summing them at intervals? Perhaps some way of determining what elements of the table have already been farmed out to the kernels?

Answer 1

You nearly gave the answer yourself, when you said "Perhaps letting each kernel have its own value for counter and summing them at intervals?".

Try something like this:

counter = 1;
SetSharedVariable[counter];
ParallelEvaluate[last = AbsoluteTime[]; localcounter = 1;]
Timing[Monitor[
  ParallelTable[localcounter++; 
    If[AbsoluteTime[] - last > 1, last = AbsoluteTime[]; 
     counter += localcounter; localcounter = 0;], {n, 10^6}];, 
  ProgressIndicator[counter, {0, 10^6}]]]

Note that it takes longer than your first single-CPU case only because it actually does something in the loop.

You can change the test AbsoluteTime[] - last > 1 to something more frequent like AbsoluteTime[] - last > 0.1.

Answer 2

This seems hard to solve. From the manual:

Unless you use shared variables, the parallel evaluations performed are completely independent and cannot influence each other. Furthermore, any side effects, such as assignments to variables, that happen as part of evaluations will be lost. The only effect of a parallel evaluation is that its result is returned at the end.

However, a rough progress indicator can still be gotten using the old Printstatement: