How to prevent a system call from being executed with ptrace

Question

I'm working on a Ideone-like system where untrusted user code must run in sandboxed mode.

For this I've been looking the possibilities of ptrace for a first layer of protection. However, after a few experiments it seems that:

• I can intercept a system call before it's called and modify the input arguments.
• I can intercept a system call after it has been called and change the return value.
• However, there seems to be no way to prevent the call from happing at all (except for killing the entire application).

I want to intercept certain system calls and return a fake result code without the call actually happening. Is there a way to implement this?

Answer 1

Please keep in mind that your sandbox can only be secure if the code it runs is not multi-threaded. You'll also have to take great care to prevent the sand-boxed code from forking as well.

See, for example, the following discussion of a paper about the issues by Robert Watson:

Exploiting races in system call wrappers

The paper is linked to in that article, but I'll offer the link here directly as well:

"Exploiting Concurrency Vulnerabilities in System Call Wrappers"

The better approach still seems to be as is recommended by Watson: integrate the security framework entirely into the kernel and take care in its use to avoid concurrency issues. Linux and NetBSD and Mac OS X and other security-oriented systems already provide such frameworks and so all that's necessary if using those systems is to implement your policies within those existing frameworks. I.e. don't even try to implement your security policies in system call wrappers or other system call interposition mechanisms.

Answer 2

you could jump the instruction that executes the system call, by incrementing the IP (instruction pointer), this way the call will not be executed and you can set the return value as usual.

Edit:

There's a ptrace wrapper called pinktrace, that should make your job easier, also some more information here:

https://security.stackexchange.com/questions/8484/wrapping-system-call-in-reliable-and-secure-way