What's the best way to transfer data from python to another application in windows?

Question

I'm developing an application with a team in .Net (C++) and provide a COM interface to interact with python and other languages.

What we've found is that pushing data through COM turns out to be pretty slow.

I've considered several alternatives:

• dumping data to a file and sending the file path through com
• Shared Memory via mmap?
• Stream data through a socket directly?

From your experience what's the best way to pass around data?

Answer 1

Staying within the Windows interprocess communication mechanisms, we had positive experience using windows named pipes. Using Windows overlapped IO and the win32pipe module from pywin32.

You can learn much about win32 and python in the Python Programming On Win32 book.

The sending part simply writes to r'\\.\pipe\mypipe'.

A listener (ovpipe) object holds an event handle, and waiting for a message with possible other events involves calling win32event.WaitForMultipleObjects.

rc = win32event.WaitForMultipleObjects(
    eventlist,    # Objects to wait for.
    0,            # Wait for one object
    timeout)      # timeout in milli-seconds.

Here is part of the python overlapped listener class:

import win32event
import pywintypes
import win32file
import win32pipe

class ovpipe:
"Overlapped I/O named pipe class"
def __init__(self):
    self.over=pywintypes.OVERLAPPED()
    evt=win32event.CreateEvent(None,1,0,None)
    self.over.hEvent=evt
    self.pname='mypipe'
    self.hpipe = win32pipe.CreateNamedPipe(
        r'\\.\pipe\mypipe',             # pipe name 
        win32pipe.PIPE_ACCESS_DUPLEX|   # read/write access
        win32file.FILE_FLAG_OVERLAPPED,
        win32pipe.PIPE_TYPE_MESSAGE|    # message-type pipe 
        win32pipe.PIPE_WAIT,            # blocking mode 
        1,                              # number of instances 
        512,                            # output buffer size 
        512,                            # input buffer size 
        2000,                           # client time-out
        None)                           # no security attributes
    self.buffer = win32file.AllocateReadBuffer(512)
    self.state='noconnected'
    self.chstate()

def execmsg(self):
    "Translate the received message"
    pass

def chstate(self):
    "Change the state of the pipe depending on current state"
    if self.state=='noconnected':
        win32pipe.ConnectNamedPipe(self.hpipe,self.over)
        self.state='connectwait'
        return -6

    elif self.state=='connectwait':
        j,self.strbuf=win32file.ReadFile(self.hpipe,self.buffer,self.over)
        self.state='readwait'
        return -6

    elif self.state=='readwait':
        size=win32file.GetOverlappedResult(self.hpipe,self.over,1)
        self.msg=self.strbuf[:size]
        ret=self.execmsg()
        self.state = 'noconnected'
        win32pipe.DisconnectNamedPipe(self.hpipe)
        return ret

Answer 2

XML/JSON and a either a Web Service or directly through a socket. It is also language and platform independent so if you decide you want to host the python portion on UNIX you can, or if you want to suddenly use Java or PHP or pretty much any other language you can.

As a general rule proprietary protocols/architectures like COM offer more restrictions than they do benefits. This is why the open specifications appeared in the first place.

HTH