Testing a Node library working with Docker containers

Question

I'm currently writing a Node library to execute untrusted code within Docker containers. It basically maintains a pool of containers running, and provides an interface to run code in one of them. Once the execution is complete, the corresponding container is destroyed and replaced by a new one.

The four main classes of the library are:

• Sandbox. Exposes a constructor with various options including the pool size, and two public methods: executeCode(code, callback) and cleanup(callback)
• Job. A class with two attributes, code and callback (to be called when the execution is complete)
• PoolManager, used by the Sandbox class to manage the pool of containers. Provides the public methods initialize(size, callback) and executeJob(job, callback). It has internal methods related to the management of the containers (_startContainer, _stopContainer, _registerContainer, etc.). It uses an instance of the dockerode library, passed in the constructor, to do all the docker related stuff per se.
• Container. A class with the attributes tmpDir, dockerodeInstance, IP and a public method executeCode(code, callback) which basically sends a HTTP POST request against ContainerIP:3000/compile along with the code to compile (a minimalist API runs inside each Docker container).

In the end, the final users of the library will only be using the Sandbox class.

---

Now, my question is: how should I test this?

First, it seems pretty clear to my that I should begin by writing functional tests against my Sandbox class:

• it should create X containers, where X is the required pool size
• it should correctly execute code (including the security aspects: handling timeouts, fork bombs, etc. which are in the library's requirements)
• it should correctly cleanup the resources it uses

But then I'm not sure what else it would make sense to test, how to do it, and if the architecture I'm using is suitable to be correctly tested.

Any idea or suggestion related to this is highly appreciated! :) And feel free to ask for a clarification if anything looks unclear.

Christophe

Answer 1

Try and separate your functional and unit testing as much as you can.

If you make a minor change to your constructor on Sandbox then I think testing will become easier. Sandbox should take a PoolManager directly. Then you can mock the PoolManager and test Sandbox in isolation, which it appears is just the creation of Jobs, calling PoolManager for Containers and cleanup. Ok, now Sandbox is unit tested.

PoolManager may be harder to unit test as the Dockerode client might be hard to mock (API is fairly big). Regardless if you mock it or not you'll want to test:

• Growing/shrinking the pool size correctly
• Testing sending more requests than available containers in the pool
• How stuck containers are handled. Both starting and stopping
• Handling of network failures (easier when you mock things)
• Retries
• Any of failure cases you can think of

The Container can be tested by firing up the API from within the tests (in a container or locally). If it's that minimal recreating it should be straightforward. Once you have that it's really just testing an HTTP client it sounds like.

The source code for the actual API within the container can be tested however you like with standard unit tests. Because you're dealing with untrusted code there are a lot of possibilities:

• Doesn't compile
• Never completes execution
• Never starts
• All sorts of bombs
• Uses all host's disk space
• Is a bot and talks over the network

The code could do basically anything. You'll have to pick the things you care about. Try and restrict everything else.

Functional tests are going to be important too, there are a log of pieces to deal with here and mocking Docker isn't going to be easy.

Code isolation is a difficult problem; I wish Docker was around last time I had to deal with it. Just remember that your customers will always do things you didn't expect! Good luck!