Custom allocation and Boehm GC

Question

In my on-again-off-again compiler project, I've implemented closures as allocated memory with an executable prefix. So a closure is allocated like this:

c = make_closure(code_ptr, env_size, env_data);

c is a pointer to a block of allocated memory, which looks like this:

movl $closure_call, %eax
call *%eax
.align 4
; size of environment
; environment data
; pointer to closure code

closure_call is a helper function that looks at the address most recently placed on the stack and uses it to find the closure data and code pointer. Boehm GC is used for general memory management, and when the closure is no longer referenced it can be deallocated by the GC.

Anyway this allocated memory needs to be marked as executable; in fact the entire pages it spans get marked. As closures are created and deallocated, more and more heap memory in the process will be executable.

For defensive programming reasons I'd prefer to minimise the amount of executable heap. My plan is to try to keep all closures together on the same page(s), and to allocate and deallocate executable pages as needed; i.e. to implement a custom allocator for closures. (This is easier if all closures are the same size; so the first step is moving the environment data into a separate non-executable allocation that can be managed normally. It also makes defensive programming sense.)

But the remaining issue is GC. Boehm already does this! What I want is to somehow tell Boehm about my custom allocations, and get Boehm to tell me when they're able to be GC'd, but to leave it up to me to deallocate them.

So my question is, are there hooks in Boehm that provide for custom allocations like this?

Answer 1

You may be able to do what you want with a finalizer - Boehm GC would still deallocate it, but you would have an opportunity beforehand to memset the closure with breakpoint ops (0xCC on x86) and mark its page non-executable if possible.

However, finalizers have a performance cost, so should not be used lightly. Boehm GC is based on a mark-sweep algorithm, which first identifies all chunks that should not be freed (mark.c), then frees everything else all at once (reclaim.c). In your case, it makes sense to modify the reclamation process to also fill all free space in your executable region with breakpoint ops, and mark pages non-executable as they become completely empty. This avoids finalizers, at the expense of forking the library (I couldn't find any extensibility mechanism for this).

Finally, note that execution prevention is a defense-in-depth measure, and should not be your only security protection. Return-oriented programming can be used to execute arbitrary code using non-modifiable executable regions.