How can I handle IPC between C and Python?

Question

I have a an application with two processes, one in C and one in Python. The C process is where all the heavy lifting is done, while the Python process handles the user interface.

The C program writes to a large-ish buffer 4 times per second, and the Python process reads this data. To this point the communication to the Python process has been done by AMQP. I would much rather setup some for of memory sharing between the two processes to reduce overhead and increase performance.

What are my options here? Ideally I would simply have the Python process read the physical memory straight (preferable from memory and not from disk), and then taking care of race conditions with Semaphores or something similar. This is however something I have little experience with, so I'd appreciate any help I can get.

I am using Linux btw.

Answer 1

This question has been asked for a long time. I believe the questioner already has the answer, so I wrote this answer for people later coming.

/*C code*/

#include <stdio.h>
#include <string.h>
#include <errno.h>
#include <sys/ipc.h>
#include <sys/shm.h>

#define GETEKYDIR ("/tmp")
#define PROJECTID  (2333)
#define SHMSIZE (1024)

void err_exit(char *buf) {
    fprintf(stderr, "%s\n", buf);
    exit(1);
}


    int
main(int argc, char **argv)
{

    key_t key = ftok(GETEKYDIR, PROJECTID);
    if ( key < 0 )
        err_exit("ftok error");

    int shmid;
    shmid = shmget(key, SHMSIZE, IPC_CREAT | IPC_EXCL | 0664);
    if ( shmid == -1 ) {
        if ( errno == EEXIST ) {
            printf("shared memeory already exist\n");
            shmid = shmget(key ,0, 0);
            printf("reference shmid = %d\n", shmid);
        } else {
            perror("errno");
            err_exit("shmget error");
        }
    }

    char *addr;

    /* Do not to specific the address to attach
     * and attach for read & write*/
    if ( (addr = shmat(shmid, 0, 0) ) == (void*)-1) {
        if (shmctl(shmid, IPC_RMID, NULL) == -1)
            err_exit("shmctl error");
        else {
            printf("Attach shared memory failed\n");
            printf("remove shared memory identifier successful\n");
        }

        err_exit("shmat error");
    }

    strcpy( addr, "Shared memory test\n" );

    printf("Enter to exit");
    getchar();

    if ( shmdt(addr) < 0) 
        err_exit("shmdt error");

    if (shmctl(shmid, IPC_RMID, NULL) == -1)
        err_exit("shmctl error");
    else {
        printf("Finally\n");
        printf("remove shared memory identifier successful\n");
    }

    return 0;
}

#python 

# Install sysv_ipc module firstly if you don't have this
import sysv_ipc as ipc

def main():
    path = "/tmp"
    key = ipc.ftok(path, 2333)
    shm = ipc.SharedMemory(key, 0, 0)

    #I found if we do not attach ourselves
    #it will attach as ReadOnly.
    shm.attach(0,0)  
    buf = shm.read(19)
    print(buf)
    shm.detach()
    pass

if __name__ == '__main__':
    main()

The C program need to be executed firstly and do not just stop it before python code executed, it will create the shared memory segment and write something into it. Then Python code attach the same segment and read data from it.

After done the all things, press enter key to stop C program and remove shared memory ID.

We can see more about SharedMemory for python in here: http://semanchuk.com/philip/sysv_ipc/#shared_memory

Answer 2

Suggestion #1: The simplest way should be using TCP. You mentioned your data size is large. Unless your data size is too huge, you should be fine using TCP. Ensure you make separate threads in C and Python for transmitting/receiving data over TCP.

Suggestion #2: Python supports wrappers over C. One popular wrapper is ctypes - http://docs.python.org/2/library/ctypes.html

Assuming you are familiar with IPC between two C programs through shared-memory, you can write a C-wrapper for your python program which reads data from the shared memory.

Also check the following diccussion which talks about IPC between python and C++: Simple IPC between C++ and Python (cross platform)