Is there a way to sandbox test execution with pytest, especially filesystem access?

Question

I'm interested in executing potentially untrusted tests with pytest in some kind of sandbox, like docker, similarly to what continuous integration services do.

I understand that to properly sandbox a python process you need OS-level isolation, like running the tests in a disposable chroot/container, but in my use case I don't need to protect against intentionally malicious code, only from dangerous behaviour of pairing "randomly" functions with arguments. So lesser strict sandboxing may still be acceptable. But I didn't find any plugin that enables any form of sandboxing.

What is the best way to sandbox tests execution in pytest?

Update: This question is not about python sandboxing in general as the tests' code is run by pytest and I can't change the way it is executed to use exec or ast or whatever. Also using pypy-sandbox is not an option unfortunately as it is "a prototype only" as per the PyPy feature page.

Update 2: Hoger Krekel on the pytest-dev mailing list suggests using a dedicated testuser via pytest-xdist for user-level isolation:

py.test --tx ssh=OTHERUSER@localhost --dist=each

which made me realise that for my CI-like use case:

having a "disposable" environment is as important as having a isolated one, so that every test or every session runs from the same initial state and it is not influenced by what older sessions might have left on folders writable by the testuser (/home/testuser, /tmp, /var/tmp, etc).

So the testuser+xdist is close to a solution, but not quite there.

Just for context I need isolation to run pytest-nodev.

Answer 1

After quite a bit of research I didn't find any ready-made way for pytest to run a project tests with OS-level isolation and in a disposable environment. Many approaches are possible and have advantages and disadvantages, but most of them have more moving parts that I would feel comfortable with.

The absolute minimal (but opinionated) approach I devised is the following:

• build a python docker image with:
  • a dedicated non-root user: pytest
  • all project dependencies from requirements.txt
  • the project installed in develop mode
• run py.test in a container that mounts the project folder on the host as the home of pytest user

To implement the approach add the following Dockerfile to the top folder of the project you want to test next to the requirements.txt and setup.py files:

FROM python:3

# setup pytest user
RUN adduser --disabled-password --gecos "" --uid 7357 pytest
COPY ./ /home/pytest
WORKDIR /home/pytest

# setup the python and pytest environments
RUN pip install --upgrade pip setuptools pytest
RUN pip install --upgrade -r requirements.txt
RUN python setup.py develop

# setup entry point
USER pytest
ENTRYPOINT ["py.test"]

Build the image once with:

docker build -t pytest .

Run py.test inside the container mounting the project folder as volume on /home/pytest with:

docker run --rm -it -v `pwd`:/home/pytest pytest [USUAL_PYTEST_OPTIONS]

Note that -v mounts the volume as uid 1000 so host files are not writable by the pytest user with uid forced to 7357.

Now you should be able to develop and test your project with OS-level isolation.

Update: If you also run the test on the host you may need to remove the python and pytest caches that are not writable inside the container. On the host run:

rm -rf .cache/ && find . -name __pycache__  | xargs rm -rf

Answer 2

To be honest, this seems like a great use case for something like docker. Sure, you're not handling it completely cleanly using only python, but you can abuse the host OS to your heart's desire and not have to worry about long-term damage. Plus, unlike a lot of CI solutions, it can run comfortably on your development machine.

Also note that whether or not your code is intentionally malicious or not, having that sort of isolation is still beneficial to prevent accidents like:

rm -rf /usr/local/share/ myapp