Passing a large data structure over dbus

Question

I'm using dbus to communicate two programs. One creates a large image and it later sends it other program for further processing. I'm passing the image as ByteArray.

With 2000x2000 images my program works, but with 4000x4000 it crasses with:

process 2283: arguments to dbus_message_iter_append_fixed_array() were       
incorrect,assertion "n_elements <= DBUS_MAXIMUM_ARRAY_LENGTH / _dbus_type_get_alignment  
(element_type)" failed in file dbus-message.c line 2628.

I understand that this means that I'm passing an array larger than allowed. Is there other way of passing large data strucutures in dbus?

This is an excerpt of the code I'm using:

handle = StringIO()
# hdulist contains the large data structure
hdulist.writeto(handle)
hdub = dbus.ByteArray(handle.getvalue())
# this sends the image via dbus
self.dbi.store_image(hdub)

In the other end I have something like

def store_image(self, bindata):
    # Convert binary data back to HDUList
    handle = StringIO.StringIO(bindata)
    hdulist = pyfits.open(handle)

Answer 1

I don't think Dbus is really the best way to send large amounts of data.

How about writing the data structure out to a file in /tmp, and just passing the filename between the programs via dbus instead?

Answer 2

D-bus won't let you pass more than 128Mb of data per message, and the limit is usually set to even lower value in /etc/dbus-1/session.conf

A named pipe is what you're looking for. It's like a temporary file except that the data is not written to disk, but streamed from one process to another with very low overhead. It has no limits on data size. See http://en.wikipedia.org/wiki/Named_pipe and Create a temporary FIFO (named pipe) in Python? for details.