Ruby C extensions API questions

Question

So, recently I had the unfortunate need to make a C extension for Ruby (because of performance). Since I was having problems with understanding VALUE (and still do), so I looked into the Ruby source and found: typedef unsigned long VALUE; (Link to Source, but you will notice that there are a few other 'ways' it's done, but I think it's essentially a long; correct me if I'm wrong). So, while investigating this further I found an interesting blog post, which says:

"...in some cases the VALUE object could BE the data instead of POINTING TO the data."

What confuses me is that, when I attempt to pass a string to C from Ruby, and use RSTRING_PTR(); on the VALUE (passed to the C-function from Ruby), and try to 'debug' it with strlen(); it returns 4. Always 4.

example code:

VALUE test(VALUE inp) {
    unsigned char* c = RSTRING_PTR(inp);
    //return rb_str_new2(c); //this returns some random gibberish
    return INT2FIX(strlen(c));
}

This example returns always 1 as the string length:

VALUE test(VALUE inp) {
    unsigned char* c = (unsigned char*) inp;
    //return rb_str_new2(c); // Always "\x03" in Ruby.
    return INT2FIX(strlen(c));
}

Sometimes in ruby I see an Exception saying "Can't convert Module to String" (or something along those lines, however I was messing with the code so much trying to figure this out that I am unable to reproduce the error now the error would happen when I tried StringValuePtr(); [I'm a bit unclear what this exactly does. Documentation says it changes the passed paramater to char*] on inp):

VALUE test(VALUE inp) {
    StringValuePtr(inp);
    return rb_str_new2((char*)inp); //Without the cast, I would get compiler warnings
} 

So, the Ruby code in question is: MyMod::test("blahblablah")

EDIT: Fixed a few typos and updated the post a little.

---

The questions

1. What exactly does VALUE imp hold? A pointer to the object/value? The value itself?
2. If it holds the value itself: when does it do that, and is there a way to check for it?
3. How do I actually access the value (since I seem to accessing almost everything but the value)?

P.S: My understanding of C isn't really the best, but it's a work in progress; also, read the comments in the code snippets for some additional description (if it helps).

Thanks!

Answer 1

Ruby Strings vs. C strings

Let's start with strings first. First of all, before trying to retrieve a string in C, it is good habit to call StringValue(obj) on your VALUE first. This ensures that you will really deal with a Ruby string in the end because if it is not already a string, then it will turn it into one by coercing it with a call to that object's to_str method. So this makes things safer and prevents the occasional segfault you might get otherwise.

The next thing to watch out for is that Ruby strings are not \0-terminated as your C code would expect them to make things like strlen etc. work as expected. Ruby's strings carry their length information with them instead - that's why in addition to RSTRING_PTR(str) there is also the RSTRING_LEN(str) macro to determine the actual length.

So what StringValuePtr now does is returning the non-zero-terminated char * to you - this is great for buffers where you have a separate length, but not what you want for e.g. strlen. Use StringValueCStr instead, it will modify the string to be zero-terminated so that it is safe for usage with functions in C that expect it to be zero-terminated. But, try to avoid this wherever possible, because this modification is much less performant than retrieving the non-zero-terminated string that does not have to be modified at all. It's surprising if you keep an eye on this how rarely you will actually need "real" C strings.

self as an implicit VALUE argument

Another reason why your current code doesn't work as expected is that every C function to be called by Ruby gets passed self as an implicit VALUE.

• No arguments in Ruby ( e.g. obj.doit ) translates to

  VALUE doit(VALUE self)

• Fixed amount of arguments (>0, e.g. obj.doit(a, b)) translates to

  VALUE doit(VALUE self, VALUE a, VALUE b)

• Var args in Ruby ( e.g. obj.doit(a, b=nil)) translates to

  VALUE doit(int argc, VALUE *argv, VALUE self)

in Ruby. So what you were working on in your example is not the string passed to you by Ruby but actually the current value of self, that is the object that was the receiver when you called that function. A correct definition for your example would be

static VALUE test(VALUE self, VALUE input) 

I made it static to point out another rule that you should follow in your C extensions. Make your C functions only public if you intend to share them among several source files. Since that's almost never the case for function that you attach to a Ruby class, you should declare them as static by default and only make them public if there is a good reason to do so.

What is VALUE and where does it come from?

Now to the harder part. If you dig down deeply into Ruby internals, then you will find the function rb_objnew in gc.c. Here you can see that any newly created Ruby object becomes a VALUEby being cast as one from something called the freelist. It's defined as:

#define freelist objspace->heap.freelist

You can imagine the objspace as a huge map that stores each and every object that is currently alive at a given point in time in your code. This is also where the garbage collector fulfills his duty and the heap struct in particular is the place where new objects are born. The "freelist" of the heap is again declared as being an RVALUE *. This is the C-internal representation of the Ruby built-in types. An RVALUE is actually defined as follows:

typedef struct RVALUE {
    union {
    struct {
        VALUE flags;        /* always 0 for freed obj */
        struct RVALUE *next;
    } free;
    struct RBasic  basic;
    struct RObject object;
    struct RClass  klass;
    struct RFloat  flonum;
    struct RString string;
    struct RArray  array;
    struct RRegexp regexp;
    struct RHash   hash;
    struct RData   data;
    struct RTypedData   typeddata;
    struct RStruct rstruct;
    struct RBignum bignum;
    struct RFile   file;
    struct RNode   node;
    struct RMatch  match;
    struct RRational rational;
    struct RComplex complex;
    } as;
    #ifdef GC_DEBUG
    const char *file;
    int   line;
    #endif
} RVALUE;

That is, basically a union of core data types that Ruby knows about. Missing something? Yes, Fixnums, Symbols, nil and boolean values are not included there. It's because these kinds of objects are directly represented using the unsigned long that a VALUE boils down to in the end. I think the design decision there was (besides being a cool idea) that dereferencing a pointer might be slightly less performant than the bit shifts that are currently needed when transforming the VALUE to what it actually represents. Essentially

obj = (VALUE)freelist;

says give me whatever freelist points to currently and treat is as unsigned long. This is safe because freelist is a pointer to an RVALUE - and a pointer can also be safely interpreted as unsigned long. This implies that every VALUE except those carrying Fixnums, symbols, nil or Booleans are essentially pointers to an RVALUE, the others are directly represented within the VALUE.

Your last question, how can you check for what a VALUE stands for? You can use the TYPE(x) macro to check whether a VALUE's type would be one of the "primitive" ones.