Technique for using std::ifstream, std::ofstream in python via SWIG?

Question

Is there a way to use std::[io]fstream's in python via swig?

I have a c-class with functions like:

void readFrom(std::istream& istr);
void writeTo(std::ostream& ostr);

I would like to construct, in python, an std::ofstream instance and pass it in as the argument to writeTo (and do the same thing for reading).

I tried making a function like

std::ostream& make_ostream(const std::string& file_name){
    return std::ofstream( file_name.c_str() );
}

inside the swig .i file, so that this function would be part of the interface. However this doesn't work. There is a problem since the stream classes are non-copyable.

Although std_iostream.i seems to help with using the generic [io]stream classes, it doesn't help with making the file streams that I need.

Answer 1

My preferred solution to this problem would be to make the interface exposed to Python developers as "Pythonic" as possible. In this instance that would be to accept python file objects as your ostream and istream arguments.

To achieve that we have to write a typemap to set up each mapping.

I've written the following header file to demonstrate this in action:

#ifndef TEST_HH
#define TEST_HH
#include <iosfwd>

void readFrom(std::istream& istr);
void writeTo(std::ostream& ostr);
#endif

Which I wrote a dummy implementation for testing as:

#include <iostream>
#include <cassert>
#include "test.hh"

void readFrom(std::istream& istr) {
  assert(istr.good());
  std::cout << istr.rdbuf() << "\n";
}

void writeTo(std::ostream& ostr) {
  assert(ostr.good());
  ostr << "Hello" << std::endl;
  assert(ostr.good());
}

With that in place I was able to wrap it successfully using:

%module test

%{
#include <stdio.h>
#include <boost/iostreams/stream.hpp>
#include <boost/iostreams/device/file_descriptor.hpp>
namespace io = boost::iostreams;
typedef io::stream_buffer<io::file_descriptor_sink> boost_ofdstream;
typedef io::stream_buffer<io::file_descriptor_source> boost_ifdstream;
%}

%typemap(in) std::ostream& (boost_ofdstream *stream=NULL) {
  int fd = -1;

  #if PY_VERSION_HEX >= 0x03000000
  fd = PyObject_AsFileDescriptor($input);
  #else 
  FILE *f=PyFile_AsFile($input); // Verify the semantics of this
  if (f) fd = fileno(f);
  #endif
  if (fd < 0) {
    SWIG_Error(SWIG_TypeError, "File object expected.");
    SWIG_fail;
  }
  else {
    // If threaded incrment the use count
    stream = new boost_ofdstream(fd, io::never_close_handle);
    $1 = new std::ostream(stream);
  }
}

%typemap(in) std::istream& (boost_ifdstream *stream=NULL) {
  int fd = -1;

  #if PY_VERSION_HEX >= 0x03000000
  fd = PyObject_AsFileDescriptor($input);
  #else 
  FILE *f=PyFile_AsFile($input); // Verify the semantics of this
  if (f) fd = fileno(f);
  #endif
  if (fd < 0) {
    SWIG_Error(SWIG_TypeError, "File object expected.");  
    SWIG_fail;
  }
  else {
    stream = new boost_ifdstream(fd, io::never_close_handle);
    $1 = new std::istream(stream);
  }
}

%typemap(freearg) std::ostream& {
  delete $1;
  delete stream$argnum;
}

%typemap(freearg) std::istream& {
  delete $1;
  delete stream$argnum;
}

%{
#include "test.hh"
%}
%include "test.hh"

The core bit of this is basically calling PyFile_AsFile() to get a FILE* from the Python file object. With that we can then construct a boost object that uses a file descriptor as the source/sink as appropriate.

The only thing that remains is to clean up the objects we created after the call has happened (or if an error prevented the call from happening).

With that in place we can then use it as expected from within Python:

import test
outf=open("out.txt", "w")
inf=open("in.txt", "r")

outf.write("Python\n");

test.writeTo(outf)
test.readFrom(inf)

outf.close()
inf.close()

Note the buffering semantics might not produce the results you expected, for instance in out.txt I get:

Hello
Python

which is the opposite order of the calls. We can fix that also by forcing a call to file.flush() on the Python file object in our typemap, before constructing a C++ stream:

%typemap(in) std::ostream& (boost_ofdstream *stream=NULL) {
  PyObject_CallMethod($input, "flush", NULL);
  FILE *f=PyFile_AsFile($input); // Verify the semantics of this
  if (!f) {
    SWIG_Error(SWIG_TypeError, "File object expected.");
    SWIG_fail;
  }
  else {
    // If threaded incrment the use count
    stream = new boost_ofdstream(fileno(f), io::never_close_handle);
    $1 = new std::ostream(stream);
  }
}

Which has the desired behaviour.

Other notes:

1. If you've got multithread code and the C++ calls are happening without the GIL you'll need to call PyFile_IncUseCount and PyFile_DecUseCount in the in and freearg typemaps respectively to make sure that nothing can close the file whilst you're still using it.
2. I've assumed that PyFile_AsFile returns NULL if the object it's given isn't a file - the documentation doesn't seem to specify that either way, so you could use PyFile_Check to be sure.
3. If you wanted to be super flexible you could accept strings from Python and construct a std::ifstream as appropriate using PyString_Check/PyFile_Check to decide which action to take in the typemap.
4. Some C++ standard libraries provide an ifstream/ofstream constructor which takes FILE*, as an extension. If you have one of those you could use it instead of relying on boost.

Answer 2

I don't know swig but assuming you need to create a copyable object, you might get away using a function like

std::shared_ptr<std::ostream> make_ostream(std::string const& filename) {
    return std::make_shared<std::ofstream>(filename);
}

... and then use a forwarding function to call the function you actually want to call:

void writeTo(std::shared_ptr<std::ostream> stream) {
    if (stream) {
        writeTo(*stream);
    }
}

(if overloading the names causes issues, you could call the forwarding function differently, of course).

Answer 3

I ended up just writing my own proxy class for use within the interface. So I used SWIG to wrap this class:

 /**
 * Simple class to expose std::streams in the python
 * interface.  works around some issues with trying to directy
 * the file stream objects
 */
class ifstream_proxy: boost::noncopyable{
    public:
        ifstream_proxy(): m_istr(){
            // no op
        }
        virtual ~ifstream_proxy(){
            // no op
        }
        void open(const std::string& fname ){
            m_istr.close(); 
            m_istr.open( fname.c_str(), std::ifstream::in|std::ifstream::binary) ;
        }
        std::istream& stream(){
            return m_istr;
        }
        // TBD: do I want to  add additional stream manipulation functions?
   private: 
        std::ifstream m_istr;
};

and in python call make the calls

>>> proxy=ifstream_proxy()
>>> proxy.open('file_to_read_from.txt')
>>> readFrom( stream_proxy.stream() )