python shutil.rmtree throwing errors

Question

Can someone explain to me why shutil.rmtree is throwning errors saying directory is not empty?

Traceback (most recent call last):
  File "C:\Python27\lib\threading.py", line 810, in __bootstrap_inner
    self.run()
  File "C:\Python27\lib\threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "W:\__init__.py", line 90, in makePatch
    myprog.copy_data()
  File "W:\myprog.py", line 143, in copy_data
    self.cleanupTempDir()
  File "W:\myprog.py", line 138, in cleanupTempDir
    shutil.rmtree(self.TEMP_DIR)
  File "C:\Python27\lib\shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "C:\Python27\lib\shutil.py", line 256, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "C:\Python27\lib\shutil.py", line 254, in rmtree
    os.rmdir(path)
WindowsError: [Error 145] The directory is not empty: u'e:\\PatchData\\Data'

Answer 1

Linux: I had encountered this error as well while handling a temporary folder: I had a os.makedirs call within a __init__ constructor and a shutil.rmtree call within a __del__ deconstructor, the code made use of the multiprocessing library; the folder was constructed sequentially but destroyed whenever the process terminated is execution. As such the error was not always reproducible, but it occurred, supposedly, only when two processes tried to destroy the folder at the same time, and without a proper locking mechanism, those errors were thrown. It happened because for some unknown reason the __del__ deconstructor was called more than once.

The errors were like:

Exception ignored in: <function VideoMaker.__del__ at 0x7f6066a888b0>
Traceback (most recent call last):
  File "VideoMaker.py", line 49, in __del__
    onerror(os.unlink, fullname, sys.exc_info())
  File "/home/Marco/miniconda3/lib/python3.8/shutil.py", line 670, in _rmtree_safe_fd
    onerror(os.rmdir, path, sys.exc_info())
  File "/home/Marco/miniconda3/lib/python3.8/shutil.py", line 717, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
    shutil.rmtree(self.temp_videoclip_dir)    
  File "/home/Marco/miniconda3/lib/python3.8/shutil.py", line 672, in _rmtree_safe_fd
  File "/home/Marco/miniconda3/lib/python3.8/shutil.py", line 715, in rmtree
    onerror(os.unlink, fullname, sys.exc_info())
  File "/home/Marco/miniconda3/lib/python3.8/shutil.py", line 670, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
FileNotFoundError: [Errno 2] No such file or directory: '12.mp4'

Or

OSError: [Errno 39] Directory not empty: 'tmp/tmp.videoclips/'

---

Do not use rmtree in a __del__ deconstructor:

An object deconstructor may not always have a predictable behaviour, as such refrain of using OS' IO operations inside. Create a separate function instead and call it at the end of the object scope.

If using multiprocessing or threading : LOCK

Docs:

• multiprocessing.Lock
• threading.Lock

Docs Example: Synchronization between processes

l.acquire()
try:
    if os.path.isdir(path_to_delete):
        shutil.rmtree(path_to_delete)
finally:
    l.release()

---

If there is really no other way:

def try_to_rmtree(can_still_retry):
    if can_still_retry:
        try:
            if os.path.isdir(path_to_delete):
                shutil.rmtree(path_to_delete)
        except:
            try_to_rmtree(can_still_retry -1)

can_still_retry = 5
try_to_rmtree(can_still_retry)