Sandbox against malicious code in a Java application

Question

In a simulation server environment where users are allowed to submit their own code to be run by the server, it would clearly be advantageous for any user-submitted code to be run in side a sandbox, not unlike Applets are within a browser. I wanted to be able to leverage the JVM itself, rather than adding another VM layer to isolate these submitted components.

This kind of limitation appears to be possible using the existing Java sandbox model, but is there a dynamic way to enable that for just the user-submitted parts of a running application?

Answer 1

1. Run the untrusted code in its own thread. This for example prevents problems with infinite loops and such, and makes the future steps easier. Have the main thread wait for the thread to finish, and if takes too long, kill it with Thread.stop. Thread.stop is deprecated, but since the untrusted code shouldn't have access to any resources, it would be safe to kill it.

2. Set a SecurityManager on that Thread. Create a subclass of SecurityManager which overrides checkPermission(Permission perm) to simply throw a SecurityException for all permissions except a select few. There's a list of methods and the permissions they require here: Permissions in the Java^TM 6 SDK.

3. Use a custom ClassLoader to load the untrusted code. Your class loader would get called for all classes which the untrusted code uses, so you can do things like disable access to individual JDK classes. The thing to do is have a white-list of allowed JDK classes.

4. You might want to run the untrusted code in a separate JVM. While the previous steps would make the code safe, there's one annoying thing the isolated code can still do: allocate as much memory as it can, which causes the visible footprint of the main application to grow.

JSR 121: Application Isolation API Specification was designed to solve this, but unfortunately it doesn't have an implementation yet.

This is a pretty detailed topic, and I'm mostly writing this all off the top of my head.

But anyway, some imperfect, use-at-your-own-risk, probably buggy (pseudo) code:

ClassLoader

class MyClassLoader extends ClassLoader {   @Override   public Class<?> loadClass(String name) throws ClassNotFoundException {     if (name is white-listed JDK class) return super.loadClass(name);     return findClass(name);   }   @Override   public Class findClass(String name) {     byte[] b = loadClassData(name);     return defineClass(name, b, 0, b.length);   }   private byte[] loadClassData(String name) {     // load the untrusted class data here   } } 

SecurityManager

class MySecurityManager extends SecurityManager {   private Object secret;   public MySecurityManager(Object pass) { secret = pass; }   private void disable(Object pass) {     if (pass == secret) secret = null;   }   // ... override checkXXX method(s) here.   // Always allow them to succeed when secret==null } 

Thread

class MyIsolatedThread extends Thread {   private Object pass = new Object();   private MyClassLoader loader = new MyClassLoader();   private MySecurityManager sm = new MySecurityManager(pass);   public void run() {     SecurityManager old = System.getSecurityManager();     System.setSecurityManager(sm);     runUntrustedCode();     sm.disable(pass);     System.setSecurityManager(old);   }   private void runUntrustedCode() {     try {       // run the custom class's main method for example:       loader.loadClass("customclassname")         .getMethod("main", String[].class)         .invoke(null, new Object[]{...});     } catch (Throwable t) {}   } } 

Answer 2

Obviously such a scheme raises all sorts of security concerns. Java has a rigorous security framework, but it isn't trivial. The possibility of screwing it up and letting an unprivileged user access vital system components shouldn't be overlooked.

That warning aside, if you're taking user input in the form of source code, the first thing you need to do is compile it to Java bytecode. AFIAK, this cannot be done natively, so you'll need to make a system call to javac, and compile the source code to bytecode on disk. Here's a tutorial that can be used as a starting point for this. Edit: as I learned in the comments, you actually can compile Java code from source natively using javax.tools.JavaCompiler

Once you have JVM bytecode, you can load it into the JVM using a ClassLoader's defineClass function. To set a security context for this loaded class you will need to specify a ProtectionDomain. The minimal constructor for a ProtectionDomain requires both a CodeSource and a PermissionCollection. The PermissionCollection is the object of primary use to you here- you can use it to specify the exact permissions the loaded class has. These permissions should be ultimately enforced by the JVM's AccessController.

There's a lot of possible points of error here, and you should be extremely careful to completely understand everything before you implement anything.