Unable to open a Python subprocess in Web2py (SIGABRT)

Question

I've got an Apache2/web2py server running using the wsgi handler functionality. Within one of the controllers, I am trying to run an external executable to perform some processing on 2 files.

My approach to this is to use the subprocess module to kick off the executable. I have simplified the code to a bare-bones implementation with little success.

from subprocess import *
p = Popen(("echo", "Hello"), shell=False)
ret = p.wait()
print "Process ended with status %s" % ret

When running the above code on its own (create new file and running via python command line), it works exactly as expected.

However, as soon as I place the exact same code into my web2py controller, the external process stops working. Instead of the process returning with code 0 as is expected in the above example, it always returns -6 and "Hello" is not printed to stdout.

After doing some digging, I found that negative results from p.wait() implies that a signal caused the process to end abnormally. And, according to some docs I found, -6 corresponds to the SIGABRT signal.

I would have expected this signal to be a result of some poorly executed code in my child process. However, since this is only running echo (and since it works outside of web2py) I have my doubts that the child process is signalling itself.

Is there some web2py limitation/configuration that causes Popen() requests to always fail? If so, how can I modify my logic so that the controller (or whatever) is actually able to spawn this external process?

** EDIT: Looks like web2py applications may not like the subprocess module. According to a reply to a message reply in the web2py email group:

"You should not use subprocess in a web2py application (if you really need too, look into the admin/controllers/shell.py) but you can use it in a web2py program running from shell (web2py.py -R myprogram.py)."

I will be checking out some options based on the note here and see if any solution presents itself.

Answer 1

In the end, the best I was able to come up with involved setting up a simple XML RPC server and call the functions from that:

my_server.py

#my_server.py
from SimpleXMLRPCServer import SimpleXMLRPCServer, SimpleXMLRPCRequestHandler
from subprocess import *

proc_srvr = xmlrpclib.ServerProxy("http://localhost:12345")

def echo_fn():
    p = Popen(("echo", "hello"), shell=False)
    ret = p.wait()
    print "Process ended with status %s" % ret
    return True  # RPC Server doesn't like to return None

def main():
    server = SimpleXMLRPCServer(("localhost", 12345), ErrorHandler) 
    server.register_function(echo_fn, "echo_fn")
    while True:
        server.handle_request()

if __name__ == "__main__":
    main()

web2py_controller.py

#web2py_controller.py

def run_echo():
    proc_srvr = xmlrpclib.ServerProxy("http://localhost:12345")
    proc_srvr.echo_fn()

I'll be honest, I'm not a Python nor SimpleRPCServer guru, so the overall code may not be up to best-practice standards. However, going this route did allow me to, in effect, call a subprocess from a controller in web2py.

(Note, this was a quick and dirty simplification of the code that I have in my project. I have not validated it is in a working state, so it may require some tweaks.)