Mathematica and C/C++: Exchanging Data

Question

I would like to know how to exchange data between Mathematica and a C/C++ with pipes. In the Mathematica tutorial it says that "when you open a file or a pipe, Mathematica creates a 'stream object' that specifies the open stream associated with the file or pipe".

I know how to create files in both C and Mathematica and I can make each program read and write to them. What I still don't know how to do is how to send output from C through a pipe to another program, much less how to even do this from Mathematica.

Here is a function in which Mathematica that writes a matrix to a binary file as well as reading a file written in that format.

writeDoubleMatrix[obj_, fileName_] := Module[{file},
  file = OpenWrite[fileName, BinaryFormat -> True];
  BinaryWrite[file, Length@obj, "Integer32"];
  BinaryWrite[file, Length@obj[[1]], "Integer32"];
  BinaryWrite[file, Flatten[obj], "Real64"];
  Close[file]
 ]
readDoubleMatrix[fileName_] := Module[{file, obj, m, n},
  file = OpenRead[fileName, BinaryFormat -> True];
  m = BinaryRead[file, "Integer32"];
  n = BinaryRead[file, "Integer32"];
  obj = BinaryReadList[file, "Real64", m*n];
  Close[file];
  Partition[obj, n]
 ]

The first function will write 2 integers to a file (the size of the matrix) and the data of the matrix. I'm not doing any error checking here and thus I'm assuming that the data to be written is specifically in the form {{r11, r12, ..., r1n}, ...., {rm1, rm2, ..., rmn}}. The second function will be able to read the binary file and return the matrix.

Next comes my C program. This program will read the data stored in the file MathematicaData.bin, multiply this matrix by 2 and write data to another file.

// genData.c
#include <stdlib.h>
#include <stdio.h>

int main(int argc, char** argv){
    int m, n, i;
    double* matrix;
    FILE* fin;
    FILE* fout;

    // Reading input file
    fin = fopen(argv[1], "rb");
    fread(&m, sizeof(int), 1, fin);
    fread(&n, sizeof(int), 1, fin);
    matrix = (double*)malloc(m*n*sizeof(double));
    fread(matrix, sizeof(double), m*n, fin);
    fclose(fin);

    //Modifying data
    for (i = 0; i < m*n; ++i) matrix[i] = 2*matrix[i];

    // Writing output file
    fout = fopen(argv[2], "wb");
    fwrite(&m, sizeof(int), 1, fout);
    fwrite(&n, sizeof(int), 1, fout);
    fwrite(matrix, sizeof(double), m*n, fout);
    fclose(fout);

    // De-allocate memory used for matrix.
    free(matrix);
    return 0;
 }

This program does not have any error checking. You need to be careful how you use it otherwise the program may not detect the files or even allocate the amount of memory that you want. In any case, we can compile the program with your compiler of your choice.

gcc -o genData genData.c

And now we can try to use these functions to communicate between the two languages from Mathematica.

matrix = {{1, 2, 3, 4}, {5, 6, 7, 8}, {9, 10, 11, 12}};
writeDoubleMatrix[matrix, "MathematicaData.bin"];
Run["./genData MathematicaData.bin CData.bin"];
readDoubleMatrix["CData.bin"]

If everything went well the output you should get is

{{2., 4., 6., 8.}, {10., 12., 14., 16.}, {18., 20., 22., 24.}}

Yes, this is a very time consuming way of multiplying a matrix by 2 but this is just a simple example to show how to exchange data from Mathematica to C and from C to Mathematica. What I do not like is the fact that everything is stored to a file first and then it is read in the other program. Can someone show me how to exchange data without writing files. I have a feeling that pipes is what I need but I have no idea how to read or write them from neither language. It would be helpful if you could modify this program to adapt it to pipes.

---

UPDATE:

I found out how to make the C program "pipeable".

//genDataPipe.c
#include <stdlib.h>
#include <stdio.h>

int main(int argc, char** argv){
    int m, n, i;
    double* matrix;

    // Reading input file
    fread(&m, sizeof(int), 1, stdin);
    fread(&n, sizeof(int), 1, stdin);
    matrix = (double*)malloc(m*n*sizeof(double));
    fread(matrix, sizeof(double), m*n, stdin);

    //Modifying data
    for (i = 0; i < m*n; ++i) matrix[i] = 2*matrix[i];

    // Writing output file
    fwrite(&m, sizeof(int), 1, stdout);
    fwrite(&n, sizeof(int), 1, stdout);
    fwrite(matrix, sizeof(double), m*n, stdout);

    // Deallocate memory used for matrix.
    free(matrix);
    return 0;
}

This means that we have to use the program like this:

./genDataPipe < fileIn.bin > fileOut.bin

I have been searching in the documentation on the Mathematica side but all I have figured out that I can open a file by piping an external command. Take OpenWrite for instance:

On computer systems that support pipes, OpenWrite["!command"] runs the external program specified by command, and opens a pipe to send input to it.

This means that I can give my binary input directly to the c program. The problem is that I can't find a way of redirecting the output of the program. What I have come up with is, write a data to a file and make a wrapper to run the external command and read the contents of the output of the external command. Here we assume the existence of writeDoubleMatrix from before.

getDataPipe[fileName_] := Module[{file, obj, m, n},
  file = OpenRead["!./genDataPipe < " <> fileName, 
  BinaryFormat -> True];
  m = BinaryRead[file, "Integer32"];
  n = BinaryRead[file, "Integer32"];
  obj = BinaryReadList[file, "Real64", m*n];
  Close[file];
  Partition[obj, n]
 ]
matrix = {{1, 2, 3}, {4, 5, 6}};
writeDoubleMatrix[matrix, "MData.bin"];
output = getDataPipe["MData.bin"]

Which results in output having the following contents {{2., 4., 6.}, {8., 10., 12.}}.

The only goal that remains now is to find out how to eliminate the need of writeDoubleMatrix and pass the data directly without having to write a file.

Answer 1

The first step would be to create a named pipe, (as mentioned by Artefacto's comment above). This is called a fifo, First In First Out. A pipe makes the output from one file into the input of the other. I'd like to note that these methods are Linux only before I start.

Basically, pipes work like this:

mkfifo mypipe or in C: system ("mkfifo mypipe");

Next step is to write the output to the pipe, because everything in Linux is a file, you can just use the standard i/o operations for that, or even redirect standard input to the pipe. You already have the code for that in both Mathematica and C. So after writing the output of the file to your pipe, the Mathematica version can then read input from the pipe, multiply it, and display it in stdout or wherever you please. You really shouldn't have a problem with this method, seeing as the pipe is emptied after the reading and can be removed easily after you're done. If you want to remove the pipe afterwards, just run system ("rm myfifo");.

If you really don't want any accessory file, even though it's not that bad, try making another file that actually outputs in standard output. Then make one that let's Matematica read from the standard input. Now, to pipe through:

./cprogram | ./mprogram

This means that the output of your C program should be the input of your Mathematica program. To my knowledge, this still creates a pipe, but it will automatically be deleted by the system when it's done, and the average end user probably won't see it.