Fast recursive folder deletion - calling the right rmdir

Question

This question is not tied directly to python, but I need working implementation under python32 under windows.

Starting from this answer I assume that using shutil.rmtree() is really slow (I need to delete more than 3M files a day and it takes more than 24 hours) under windows so I wanted to use subprocess.call() and rmdir, but since I have cygwin in my %PATH% system variable wrong rmdir gets called and I'll get this:

>>> args = ['rmdir', r'D:\tmp']
>>> subprocess.call(args)
cygwin warning:
  MS-DOS style path detected: D:\tmp
  Preferred POSIX equivalent is: /cygdrive/d/tmp
  CYGWIN environment variable option "nodosfilewarning" turns off this warning.
  Consult the user's guide for more details about POSIX paths:
    http://cygwin.com/cygwin-ug-net/using.html#using-pathnames
rmdir: failed to remove `D:\\tmp': Directory not empty
1

Note: I know it's required to use /S /Q to delete folders recursively.

How can I ensure that the right rmdir is called (like under linux you would use absolute path - /bin/rm) preferably without using shell=True?

Is there alternative utility for that (something like using robocopy /MIR)?

---

Edit: speed comparison

I've tested different methods of deleting 237 GB (255,007,568,228 bytes) in 1,257,449 Files, 750,251 Folders using Measure-Command.

+-------------------+-------------+----------+-----------------+
|                   | rmdir /s /q |  shutil  | SHFileOperation |
+-------------------+-------------+----------+-----------------+
| Hours             |           3 |        5 |               6 |
| Minutes           |          26 |       52 |              14 |
| Seconds           |          46 |       13 |              48 |
| TotalMinutes      |         207 |      352 |             375 |
| TotalSeconds      |       12406 |    21134 |           22488 |
| TotalMilliseconds |    12406040 | 21133805 |        22488436 |
+-------------------+-------------+----------+-----------------+

Note: test was run on production server (so results may be affected)

Answer 1

Calling proper rmdir

I've came up with an idea of calling manually cmd.exe /C directly from %SYSTEMROOT%\System32 and clearing env variables (and it seems to work):

def native_rmdir(path):
    ''' Removes directory recursively using native rmdir command
    '''

    # Get path to cmd
    try:
        cmd_path = native_rmdir._cmd_path
    except AttributeError:
        cmd_path = os.path.join(
            os.environ['SYSTEMROOT'] if 'SYSTEMROOT' in os.environ else r'C:\Windows',
            'System32', 'cmd.exe')
        native_rmdir._cmd_path = cmd_path

    # /C - cmd will terminate after command is carried out
    # /S - recursively, 
    args = [cmd_path, '/C', 'rmdir', '/S', '/Q', path]
    subprocess.check_call(args, env={})


native_rmdir(r'D:\tmp\work with spaces')

I assume that this will work under any version of windows no matter of system-wide PATH, but I would still prefer something more "elegant".

This will delete all files it can (it won't stop after the first error).

---

Using SHFileOperation()

It's also possible to use SHFileOperation() to do this [example source]:

from win32com.shell import shell, shellcon
shell.SHFileOperation((0, shellcon.FO_DELETE, r'D:\tmp\del', None, shellcon.FOF_NO_UI))

This will stop after the first error (when I was testing this in my environment this solution tended to be slower than shutil.rmtree(), probably because UI was involved somehow).