SWIG (v1.3.29) generated C++ to Java Vector class not acting properly

Question

I have some native C++ code that I'm converting to Java using SWIG so that my Java application can use it. In particular there are some functions that return std::vector. Here's a snippet of my interface file:

%include "std_vector.i"
namespace std {
  %template(Vector) vector<double>;
  %template(Matrix) vector<vector<double> >;
}

%include "std_string.i"

std_string.i and std_vector.i were included in my build of SWIG I'm using. My first surprise was that the Java output included SWIG's "own" version of the Vector class (as opposed to using java.util.Vector). My real issue is that the Vectors that get returned from these functions do not seem to work. For example I cannot retrieve their contents using get() (sometimes crashing the program) or the size() function returning negative values. I know the Vectors contain data because I coded 'String' versions of the same functions which simply iterate through the Vectors (back in the native C++ code) and return the contents in a comma separated String value. While this is a valid workaround, ultimately I would like this to work properly with me being able to receive and manipulate the Vectors. Any helps/tips would be much appreciated.

Answer 1

The appropriate base type for wrapping std::vector in Java is java.util.AbstractList. Using java.util.Vector as a base would be odd because you'd end up with two sets of storage, one in the std::vector, and one in the java.util.Vector.

The reason SWIG doesn't do this for you though is because you can't have AbstractList<double> in Java, it has to be AbstractList<Double> (Double inherits from Object whereas double is a primitive type).

Having said all that I've put together a small example that wraps std::vector<double> and std::vector<std::vector<double> > nicely in Java. It's not complete, but it supports the "for each" style of iteration in Java and set()/get() on elements. It should be sufficient to show how to implement other things as/when you want them.

I'll talk through the interface file in sections as we go, but basically it'll all be sequential and complete.

Starting with num.i which defines our module num:

%module num

%{
#include <vector>
#include <stdexcept>

std::vector<double> testVec() {
  return std::vector<double>(10,1.0);
}

std::vector<std::vector<double> > testMat() {
  return std::vector<std::vector<double> >(10, testVec());
}
%}

%pragma(java) jniclasscode=%{
  static {
    try {
        System.loadLibrary("num");
    } catch (UnsatisfiedLinkError e) {
      System.err.println("Native code library failed to load. \n" + e);
      System.exit(1);
    }
  }
%}

We have #includes for the generated num_wrap.cxx and two implementations of functions for testing (they could be in a separate file I just put them here out of laziness/convenience).

There's also a trick there with the %pragma(java) jniclasscode= that I like to use in Java SWIG interfaces to cause the shared object/DLL to be loaded transparently for the user of the interface.

Next up in the interface file is the parts of std::vector we want to wrap. I'm not using std_vector.i because we need to make a few changes:

namespace std {

    template<class T> class vector {
      public:
        typedef size_t size_type;
        typedef T value_type;
        typedef const value_type& const_reference;
        %rename(size_impl) size;
        vector();
        vector(size_type n);
        size_type size() const;
        size_type capacity() const;
        void reserve(size_type n);
        %rename(isEmpty) empty;
        bool empty() const;
        void clear();
        void push_back(const value_type& x);
        %extend {
            const_reference get_impl(int i) throw (std::out_of_range) {
                // at will throw if needed, swig will handle
                return self->at(i);
            }
            void set_impl(int i, const value_type& val) throw (std::out_of_range) {
                // at can throw
                self->at(i) = val;
            }
        }
    };
}

The main change here is %rename(size_impl) size;, which tells SWIG to expose size() from std::vector as size_impl instead. We need to do this because Java expects size to return an int where as the std::vector version returns a size_type which more than likely won't be int.

Next up in the interface file we tell it what base class and interfaces we want to implement as well as writing some extra Java code to coerce things between functions with incompatible types:

%typemap(javabase) std::vector<double> "java.util.AbstractList<Double>"
%typemap(javainterface) std::vector<double> "java.util.RandomAccess"
%typemap(javacode) std::vector<double> %{
  public Double get(int idx) {
    return get_impl(idx);
  }
  public int size() {
    return (int)size_impl();
  }
  public Double set(int idx, Double d) {
    Double old = get_impl(idx);
    set_impl(idx, d.doubleValue());
    return old;
  }

%}

%typemap(javabase) std::vector<std::vector<double> > "java.util.AbstractList<Vector>"
%typemap(javainterface) std::vector<std::vector<double> > "java.util.RandomAccess"
%typemap(javacode) std::vector<std::vector<double> > %{
  public Vector get(int idx) {
    return get_impl(idx);
  }
  public int size() {
    return (int)size_impl();
  }
  public Vector set(int idx, Vector v) {
    Vector old = get_impl(idx);
    set_impl(idx, v);
    return old;
  }

%}

This sets a base class of java.util.AbstractList<Double> for std::vector<double> and java.util.AbstractList<Vector> for std::vector<std::vector<double> > (Vector is what we will be calling std::vector<double> on the Java side of the interface).

We also supply an implementation of get and set on the Java side which can handle the double to Double conversion and back again.

Lastly in the interface we add:

namespace std {
  %template(Vector) std::vector<double>;
  %template(Matrix) std::vector<vector<double> >;
}

std::vector<double> testVec();
std::vector<std::vector<double> > testMat();

This tells SWIG to refer to std::vector<double> (with the specific type) as Vector and similarly for std::vector<vector<double> > as Matrix. We also tell SWIG to expose our two test functions.

Next up, test.java, a simple main in Java to exercise our code a little:

import java.util.AbstractList;

public class test {
  public static void main(String[] argv) {
    Vector v = num.testVec();
    AbstractList<Double> l = v;
    for (Double d: l) {
      System.out.println(d);
    }
    Matrix m = num.testMat();
    m.get(5).set(5, new Double(5.0));
    for (Vector col: m) {
      for (Double d: col) {
        System.out.print(d + " ");
      }
      System.out.println();
    }
  }
}

To build and run this we do:

swig -java -c++ num.i
g++ -Wall -Wextra num_wrap.cxx -shared -I/usr/lib/jvm/java-6-sun/include -I/usr/lib/jvm/java-6-sun/include/linux/ -o libnum.so

javac test.java && LD_LIBRARY_PATH=. java test

I tested this with g++ version 4.4 and SWIG 1.3.40 on Linux/x86.

The complete version of num.i can be found here, but can always be reconstructed from this answer by pasting each of the part together into one file.

Things I've not implemented from AbstractList:

1. add() - can be implemented via push_back(), std_vector.i even tries to implement something compatible by default, but it doesn't work with the Double vs double problem or match the return type specified in AbstractList (Don't forget to increment modCount)
2. remove() - not great for std::vector in terms of time complexity, but not impossible to implement either (likewise with modCount)
3. A constructor which takes another Collection is recommended, but not implemented here. Can be implemented at the same place set() and get() are, but will need $javaclassname to name the generated constructor correctly.
4. You might want to use something like this to check that the size_type->int conversion in size() is sane.

Answer 2

I'm the person who offered the bounty on this question because I had the same issue. I'm a bit embarrassed to report that I've finally found the real solution -- and it's in the SWIG manual! The fix is to use the -fno-strict-aliasing flag for g++ when compiling the generated code -- simple as that. I hate to admit that it took a lot of Googling to finally found this out.

The problem is that recent versions of g++ do some aggressive optimizations that make assumptions about pointer aliasing that don't hold for the code SWIG generates for std_vector (and in other cases.) g++ 4.1 doesn't do this, but 4.4.5 definitely does. The assumptions are perfectly valid and allowed by the current ISO standard, although I'm not sure how well known they are. Basically, it's that two pointers of different types (with a few exceptions) can never point to the same address. The code that SWIG generates to convert between pointer-to-object and jlong falls afoul of this rule.