Break out of proprietary toolbox after a given time

Question

I am iterating through a large test matrix in MATLAB and calling second-party proprietary software (running in MATLAB) each time. I cannot edit the software source code. Sometimes, the software hangs, so I want to exit it after a certain amount of time and move on to the next iteration.

In pseudocode, I'm doing this:

for i = 1:n
    output(i) = proprietary_software(input(i));
end

How can I skip to the next iteration (and possibly save output(i)='too_long') if the proprietary software is taking too long?

Answer 1

You will need to call Matlab from another instance of Matlab. The other instance of Matlab will run the command and release control to the first instance of Matlab to wait while it either saves the results or reaches a certain time. In this case, it will wait 30 seconds.

You will need 1 additional function. Make sure this function is on the Matlab path.

function proprietary_software_caller(input)
    hTic=tic;
    output=proprietary_software(input);
    hToc=toc(hTic);
    if hToc<30
       save('outfile.mat','output');
    end
    exit;
end

You will need to modify your original script this way

[status,firstPID] = str2double(system('for /f "tokens=2 delims=," %F in (''tasklist /nh /fi "imagename eq Matlab.exe" /fo csv) do @echo %~F'')'));

for i = 1:n
    inputStr=num2str(input(i));
    system(['matlab.exe -nodesktop -r proprietary_software_caller\(',inputStr,'\)&']);
    hTic=tic;
    hToc=toc(hTic);
    while hToc<30 || ~(exist('outfile.mat','file')==2)
       hToc=toc(hTic);
    end
    if hToc>=30
        output(i)= 'too_long';
        [status,allPIDs]=str2double(system('for /f "tokens=2 delims=," %F in (''tasklist /nh /fi "imagename eq Matlab.exe" /fo csv) do @echo %~F'')'));
        allPIDs(allPIDs==firstPID)=[];
        for a=1:numel(allPIDs)
           [status,cmdout]=system(['taskkill /F /pid ' sprintf('%i',allPIDs(a))]);
        end
    elseif exist('outfile.mat','file')==2
        loadedData=load('outfile.mat');
        output(i)=loadedData.output;
        delete('outfile.mat');
    end
end

I hope this helps.

Answer 2

You are essentially asking for a way to implement a timeout on MATLAB code. This can be surprisingly tricky to implement. The first thing to state is that if the MATLAB code in question cannot terminate itself, either by exiting cleanly or throwing an error, then it is not possible to terminate the code without quitting or killing the MATLAB process in question. For example, throwing an error in an externally created timer does not work; the error is caught.

The first question to ask is therefore:

Can the over-running code be made to terminate itself?

This depends on the cause to the over-run, and also your access to the source code:

• If the program gets stuck in an infinite (or very long-running) loop, either in MATLAB code or a mex file for which you have source code, or which calls a user-defined callback each iteration, then you can get this code to terminate itself.
• If the program gets stuck inside a MATLAB builtin, or a p-code file or mex file for which you don't have the source code, and doesn't have support for calling a callback regularly, then it won't be possible for you to get the code to terminate itself.

Let's address the first case. The easiest way to get the code to terminate itself is to get it to throw an error, which is caught by the caller, if it exceeds the timeout time. E.g. in the OP's case:

for i = 1:n
    tic();
    try
        output(i) = proprietary_software(input(i));
    catch
    end
end

with the following code somewhere in the over-running loop, or called in a loop callback or mex file:

assert(toc() < 10, 'Timed out');

Now for the second case. You need to kill this MATLAB process, so it makes sense for this to be a MATLAB process you have spawned from your current MATLAB session. You can do this using a system call similar to this:

system('matlab -nodisplay -r code_to_run()')

While it is possible for a MATLAB process to quit itself in some situations which could be of use here (e.g. a timer function calling quit('force')), the most reliable way of killing a MATLAB process is to do it with a system call, using taskkill (Windows) or kill (Linux/Mac).

A framework using the approach of spawning and killing timed-out MATLAB processes might work like this:

1. Using system calls, launch one or more new MATLAB processes from your MATLAB session, running the code you want.
2. Use the file system or a memory mapped file to communicate between the MATLAB processes the function inputs, loop progress, outputs, process ids and timeout times.
3. Use the original MATLAB process to check the timeout times haven't been reached, or if so to terminate the process in question and instantiate a new one.
4. Use the original MATLAB process to collect up the function outputs (either from the filesystem or memory mapped file) and exit. Workers should terminate when there is no more work left

I provide a sketch only because a full working implementation of this approach is fairly involved, and in fact it has already been implemented and is publicly available in the batch_job toolbox. In the OP's case, using this toolbox (with a 10 second timeout) you'd call:

output = batch_job(@proprietary_software, input(:)', '-timeout', 10);

Note that for the toolbox to work, its root directory needs to be on your MATLAB path at startup.