Pass many pieces of data from Python to C program

Question

I have a Python script and a C program and I need to pass large quantities of data from Python script that call many times the C program. Right now I let the user choose between passing them with an ASCII file or a binary file, but both are quite slow and useless (I mean files are useful if you want to store the data, but I delete these files at the end of the script).

os.system doesn't work, the arguments are too much as the C program too uses files to return data to Python, but this is much less data.

I wonder what I can use to make this exchange fast. Writing the files to a RAM disk? If so, how can I do this?

I heard is possible to call functions from DLL using ctypes, but don't know how to compile my program as a DLL (I use wxdevc+ on Windows 7 64). Or wrap it, but still don't know if it can work and if it is efficient.

The data are vertices of a 3D mesh.

I'm running the Python script inside another program (blender (open source), and is called many times (usually more than 500 times) because it's inside a cycle. The script send vertices information (1 int index and 3 float coords) to the program, and the program should return many vertices (only int index, because I can find the corresponding vertices with Python).

So this is not interactive, it's more like a function (but it's wrote in C). The script + C program (that are add-ons of blender) that I'm writing should be cross-platform because it will be redistributed.

The program is actually wrote in C, and from Python I can know the address in memory of the struct that contains the vertices data. If only I know how to do this, should be better to pass to the C program only an address, and from there find all the other vertices (are stored in list).

But as far as I know, I can't access to the memory space of another program, and I don't know if calling the program with pipes or whatever initialize a new thread or is run inside the script (that is actually run under the Blender thread)

Here is the source and blender/source/blender/makesdna/DNA_meshdata_types.h should be the struct definition

Answer 1

Pipes are the obvious way to go; if your c program accepts input from stdin, you can use Popen. This doesn't create a "thread" as you say in your edit; it creates an entirely new process with separate memory:

from subprocess import Popen, PIPE

input = "some input"
cproc = Popen("c_prog", stdin=PIPE, stdout=PIPE)
out, err = cproc.communicate(input)

Here's a more detailed example. First, a simple c program that echoes stdin:

#include<stdio.h>
#include<stdlib.h>
#define BUFMAX 100

int main() {
    char buffer[BUFMAX + 1];
    char *bp = buffer;
    int c;
    FILE *in;
    while (EOF != (c = fgetc(stdin)) && (bp - buffer) < BUFMAX) {
        *bp++ = c;
    }
    *bp = 0;    // Null-terminate the string
    printf("%s", buffer);
}

Then a python program that pipes input (from argv in this case) to the above:

from subprocess import Popen, PIPE
from sys import argv

input = ' '.join(argv[1:])
if not input: input = "no arguments given"
cproc = Popen("./c_prog", stdin=PIPE, stdout=PIPE)
out, err = cproc.communicate(input)
print "output:", out
print "errors:", err

If you don't plan to use the c program without the python frontend, though, you might be better off inlining a c function, perhaps using instant.

from instant import inline
c_code = """
    [ ... some c code ... ] //see the below page for a more complete example.
"""
c_func = inline(c_code)

As Joe points out, you could also write a python module in c: Extending Python with C or C++

This answer discusses other ways to combine c and python: How do I connect a Python and a C program?

EDIT: Based on your edit, it sounds like you really should create a cpython extension. If you want some example code, let me know; but a full explanation would make for a unreasonably long answer. See the link above (Extending Python...) for everything you need to know.

Answer 2

If your operating system supports it, named pipes are a drop in replacement for files.