`shutil.rmtree` does not work on `tempfile.TemporaryDirectory()`

Question

Consider this test

import shutil, tempfile
from os import path
import unittest

from pathlib import Path

class TestExample(unittest.TestCase):
    def setUp(self):
        # Create a temporary directory
        self.test_dir = tempfile.TemporaryDirectory()
        self.test_dir2 = tempfile.mkdtemp()

    def tearDown(self):
        # Remove the directory after the  test
        shutil.rmtree(self.test_dir2) 
        shutil.rmtree(self.test_dir.name) #throws error

    def test_something(self):
        self.assertTrue(Path(self.test_dir.name).is_dir())
        self.assertTrue(Path(self.test_dir2).is_dir())

if __name__ == '__main__':
    unittest.main()

In tearDown however an error is raised

FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpxz7ts7a7'

which refers to self.test_dir.name.

According to the source code for tempfile, both elements are the same.

    def __init__(self, suffix=None, prefix=None, dir=None):
        self.name = mkdtemp(suffix, prefix, dir)
        self._finalizer = _weakref.finalize(
            self, self._cleanup, self.name,
            warn_message="Implicitly cleaning up {!r}".format(self))

And I'm not using it within a context, so __exit__() shouldn't be called as far as I understand.

What is happening?

Answer 1

Don't cleanup these with shutil. The tempfile.TemporaryDirectory class provides a cleanup() method, just call that if you want to opt-in to an explicit cleanup.

The reason you get the crash with your code is that the TemporaryDirectory class is designed to clean up after itself once it goes out of scope (ref count to zero). However, since you've already removed the directory from your filesystem manually, the tear down fails when the instance subsequently tries to delete itself. The "No such file or directory" error is from TemporaryDirectory's own tear down, it's not from your shutil.rmtree line!

Answer 2

It's not context related:

import tempfile,os

t = tempfile.TemporaryDirectory()
s = t.name
print(os.path.isdir(s))
# os.rmdir(s) called here triggers error on the next line
t = None
print(os.path.isdir(s))

it prints

True
False

So as soon as the reference of t is set to None the object is garbage collected and the directory is removed, as the documentation states:

On completion of the context or destruction of the temporary directory object the newly created temporary directory and all its contents are removed from the filesystem.

Uncommenting os.rmdir(s) in the snippet below throws exception when object is finalized:

Exception ignored in: <finalize object at 0x20b20f0; dead>
Traceback (most recent call last):
  File "L:\Python34\lib\weakref.py", line 519, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "L:\Python34\lib\tempfile.py", line 698, in _cleanup
    _shutil.rmtree(name)
  File "L:\Python34\lib\shutil.py", line 482, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "L:\Python34\lib\shutil.py", line 364, in _rmtree_unsafe
    onerror(os.listdir, path, sys.exc_info())
  File "L:\Python34\lib\shutil.py", line 362, in _rmtree_unsafe
    names = os.listdir(path)

So your call probably succeeds, but you get the exception at the finalization of the object (just afterwards)

Calling cleanup() object method instead of rmtree solves the issue, because the object internal state is updated for not to try to remove the directory when finalized (if you ask me, the object should test if directory exists before trying to clean it up, but even that doesn't always work since it's not an atomic operation)

So replace

shutil.rmtree(self.test_dir.name)

by

self.test_dir.cleanup()

or by nothing at all, let the object clean the directory on deletion.