Avoid Segfault in C++ code if user redefines initialize() in Ruby with Rice

Question

One problem I am struggling with while writing a C++ extension for Ruby is to make it really safe even if the user does silly things. He should get exceptions then, but never a SegFault. A concrete problem is the following: My C++ class has a non-trivial constructor. Then I use the Rice API to wrap my C++ class. If the user redefines initialize() in his Ruby code, then the initialize() function created by Rice is overwritten and the object is neither allocated nor initialized. One toy example could be the following:

class Person {
public:
  Person(const string& name): m_name (name) {}
  const string& name() const { return m_name; }
private:
  string m_name;
}

Then I create the Ruby class like this:

define_class<Person>("Person")
  .define_constructor(Constructor<Person, const string&>(), Arg("name"))
  .define_method("name", &Person::name);

Then the following Ruby Code causes a Segfault

require 'MyExtension'
class Person
  def initialize
  end
end
p = Person.new
puts p.name

There would be two possibilities I would be happy about: Forbid overwriting the initialize function in Ruby somehow or check in C++, if the Object has been allocated correctly and if not, throw an exception.

I once used the Ruby C API directly and then it was easy. I just allocated a dummy object consisting of a Null Pointer and a flag that is set to false in the allocate() function and in the initialize method, I allocated the real object and set the flag to true. In every method, I checked for that flag and raised an exception, if it was false. However, I wrote a lot of stupid repetitive code with the Ruby C API, I first had to wrap my C++ classes such that they were accessible from C and then wrap and unwrap Ruby types etc, additionally I had to check for this stupid flag in every single method, so I migrated to Rice, which is really nice and I am very glad about that.

In Rice, however, the programmer can only provide a constructor which is called in the initialize() function created by rice and the allocate() function is predefined and does nothing. I don't think there is an easy way to change this or provide an own allocate function in an "official" way. Of course, I could still use the C API to define the allocate function, so I tried to mix the C API and Rice somehow, but then I got really nasty, I got strange SegFaults and it was really ugly, so I abandoned that idea.

Does anyone here have experiences with Rice or does anyone know how to make this safe?

Answer 1

How about this

class Person
  def initialize
    puts "old"
  end
  alias_method :original_initialize, :initialize

  def self.method_added(n)
    if n == :initialize && !@adding_initialize_method
      method_name = "new_initialize_#{Time.now.to_i}"
      alias_method method_name, :initialize
      begin
        @adding_initialize_method = true
        define_method :initialize do |*args|
          original_initialize(*args)
          send method_name, *args
        end
      ensure
        @adding_initialize_method = false
      end
    end
  end
end

class Person
  def initialize
    puts "new"
  end
end

Then calling Person.new outputs

old
new

i.e. our old initialize method is still getting called

This uses the method_added hook which is called whenever a method is added (or redefined) at this point the new method already exists so it's too late to stop them from doing it. Instead we alias the freshly defined initialize method (you might want to work a little harder to ensure the method name is unique) and define another initialize that calls the old initialize method first and then the new one.

If the person is sensible and calls super from their initialize then this would result in your original initialize method being called twice - you might need to guard against this

You could just throw an exception from method_added to warn the user that they are doing a bad thing, but this doesn't stop the method from being added: the class is now in an unstable state. You could of course realias your original initialize method on top of their one.

Answer 2

In your comment you say that in the c++ code, this is a null pointer. If it is possible to call a c++ class that way from ruby, I'm afraid there is no real solution. C++ is not designed to be fool-proof. Basically this happens in c++;

Person * p = 0;
p->name();

A good c++ compiler will stop you from doing this, but you can always rewrite it in a way the compiler cannot detect what is happening. This results in undefined behaviour, the program can do anything, including crash.

Of course you can check for this in every non-static function;

const string& Person::name() const 
{ 
    if (!this) throw "object not allocated";
    return m_name; 
}

To make it easier and avoid double code, create a #define;

#define CHECK if (!this) { throw "object not allocated"; }

const string& name() const { CHECK; return m_name; }
int age() const { CHECK; return m_age; }

However it would have been better to avoid in ruby that the user can redefine initialize.