Running Python script from Cocoa application using GCD

Question

I'm trying to run a Python script from a Cocoa app. It's working just fine on the main thread, but I'd like to have it running in the background, on a concurrent GCD queue.

I'm using the following method to setup a manager class that runs the Python script:

- (BOOL)setupPythonEnvironment {
    if (Py_IsInitialized()) return YES;

    Py_SetProgramName("/usr/bin/python");
    Py_Initialize();

    NSString *scriptPath = [[NSBundle mainBundle] pathForResource:@"MyScript"     ofType:@"py"];

    FILE *mainFile = fopen([scriptPath UTF8String], "r");
    return (PyRun_SimpleFile(mainFile, (char *)[[scriptPath lastPathComponent] UTF8String]) == 0);
}

After which the script is (repeatedly) called from the following instance method, using a shared singleton instance of the manager class:

- (id)runScriptWithArguments:(NSArray *)arguments {
    return [NSClassFromString(@"MyScriptExecutor") runWithArguments:arguments];
}

The above Objective-C code hooks into the following Python code:

from Foundation import *

def run_with_arguments(arguments):
#    ...a long-running script

class MyScriptExecutor(NSObject):
    @classmethod
    def runWithArguments_(self, arguments):
        return run_with_arguments(arguments)

This works when I always run the above Objective-C methods from the main queue, but the script returns null when run from any other queue. Could someone explain me if what I'm trying to do is just not supported, and whether there's a good way around it?

The Python scripts is called often and runs long, so doing that on the main thread would be too slow, a would be running it form a serial queue. In addition, I'd like to contain the concurrency code within Objective-C as much as possible.

Thanks,

Answer 1

From this page, it looks like there are some some pretty complex threading concerns specific to embedding python. Is there a reason you couldn't just run these scripts in a separate process? For instance, the following -runBunchOfScripts method would run the script ten times (by calling -runPythonScript) on a parallel background queue, collecting the resulting outputs into an array of strings, and then calling your object back on the main thread once all the scripts has completed:

- (NSString*)runPythonScript
{
    NSTask* task = [[[NSTask alloc] init] autorelease];
    task.launchPath = @"/usr/bin/python";  
    NSString *scriptPath = [[NSBundle mainBundle] pathForResource:@"MyScript" ofType:@"py"];
    task.arguments = [NSArray arrayWithObjects: scriptPath, nil];

    // NSLog breaks if we don't do this...
    [task setStandardInput: [NSPipe pipe]];

    NSPipe *stdOutPipe = nil;
    stdOutPipe = [NSPipe pipe];
    [task setStandardOutput:stdOutPipe];

    NSPipe* stdErrPipe = nil;
    stdErrPipe = [NSPipe pipe];
    [task setStandardError: stdErrPipe];

    [task launch];        

    NSData* data = [[stdOutPipe fileHandleForReading] readDataToEndOfFile];

    [task waitUntilExit];

    NSInteger exitCode = task.terminationStatus;

    if (exitCode != 0)
    {
        NSLog(@"Error!");
        return nil;
    }

    return [[[NSString alloc] initWithBytes: data.bytes length:data.length encoding: NSUTF8StringEncoding] autorelease];
}

- (void)runBunchOfScripts
{
    dispatch_group_t group = dispatch_group_create();
    NSMutableArray* results = [[NSMutableArray alloc] init];
    for (NSUInteger i = 0; i < 10; i++)
    {
        dispatch_group_async(group, dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
            NSString* result = [self runPythonScript];
            @synchronized(results)
            {
                [results addObject: result];
            }
        });
    }

    dispatch_group_notify(group, dispatch_get_main_queue(), ^{
        [self scriptsDidFinishWithResults: results];
        dispatch_release(group);
        [results release];
    });
}

- (void)scriptsDidFinishWithResults: (NSArray*)results
{
    NSLog(@"Do something with the results...");
}

Naturally the approach of using separate processes has it's limitations, not the least of which being the hard limit on the number of processes you can launch, but it seems a lot less fraught with peril than embedding the entire interpreter. I would say that unless you need to interact chattily between the scripts and the hosting environment, this would be a better approach.