(distutil + shutil) * copytree =?

Question

I'd like the ignore pattern that shutil's copytree function provides. And I want the src tree to replace all existent files/folders in the dest directory like distutil.dir_util.copy_tree.

I'm a fairly new Python user, and can't seem to find any posts on this.

Answer 1

I was going to say this earler but I didn't. http://docs.python.org/library/shutil.html has a version of the copytree function. I looked at it to see if it would just replace existing files and from what I could tell it would overwrite existing files, but it fails if any of the directories already exist. Due to os.mkdirs failing if the directories already exist.

The needed imports:

import os
import os.path
import shutil

taking _mkdir from http://code.activestate.com/recipes/82465-a-friendly-mkdir/ (a commenter over there mentions that os.mkdirs has most of the same behavior but doesn't notice that _mkdir doesn't fail if any of the directories to be made already exist)

def _mkdir(newdir):
    """works the way a good mkdir should :)
        - already exists, silently complete
        - regular file in the way, raise an exception
        - parent directory(ies) does not exist, make them as well
    """
    if os.path.isdir(newdir):
        pass
    elif os.path.isfile(newdir):
        raise OSError("a file with the same name as the desired " \
                      "dir, '%s', already exists." % newdir)
    else:
        head, tail = os.path.split(newdir)
        if head and not os.path.isdir(head):
            _mkdir(head)
        #print "_mkdir %s" % repr(newdir)
        if tail:
            os.mkdir(newdir)

Although it doesn't take a mode argument like os.mkdirs, copytree doesn't use that so it isn't needed.

And then change copytree to call _mkdir instead of os.mkdirs:

def copytree(src, dst, symlinks=False):
    """Recursively copy a directory tree using copy2().

    The destination directory must not already exist.
    If exception(s) occur, an Error is raised with a list of reasons.

    If the optional symlinks flag is true, symbolic links in the
    source tree result in symbolic links in the destination tree; if
    it is false, the contents of the files pointed to by symbolic
    links are copied.

    XXX Consider this example code rather than the ultimate tool.

    """
    names = os.listdir(src)
    # os.makedirs(dst)
    _mkdir(dst) # XXX
    errors = []
    for name in names:
        srcname = os.path.join(src, name)
        dstname = os.path.join(dst, name)
        try:
            if symlinks and os.path.islink(srcname):
                linkto = os.readlink(srcname)
                os.symlink(linkto, dstname)
            elif os.path.isdir(srcname):
                copytree(srcname, dstname, symlinks)
            else:
                shutil.copy2(srcname, dstname)
            # XXX What about devices, sockets etc.?
        except (IOError, os.error), why:
            errors.append((srcname, dstname, str(why)))
        # catch the Error from the recursive copytree so that we can
        # continue with other files
        except Error, err:
            errors.extend(err.args[0])
    try:
        shutil.copystat(src, dst)
    except WindowsError:
        # can't copy file access times on Windows
        pass

Answer 2

Just extending Dan's answer by incorporating the ignore argument and adding shutil to the 'Error' class (for copytree):

def copytree(src, dst, symlinks=False, ignore=None):
    """Recursively copy a directory tree using copy2().

    The destination directory must not already exist.
    If exception(s) occur, an Error is raised with a list of reasons.

    If the optional symlinks flag is true, symbolic links in the
    source tree result in symbolic links in the destination tree; if
    it is false, the contents of the files pointed to by symbolic
    links are copied.

    XXX Consider this example code rather than the ultimate tool.

    """
    names = os.listdir(src)
    if ignore is not None:
        ignored_names = ignore(src, names)
    else:
        ignored_names = set()

    _mkdir(dst) # XXX
    errors = []
    for name in names:
        if name in ignored_names:
            continue
        srcname = os.path.join(src, name)
        dstname = os.path.join(dst, name)
        try:
            if symlinks and os.path.islink(srcname):
                linkto = os.readlink(srcname)
                os.symlink(linkto, dstname)
            elif os.path.isdir(srcname):
                copytree(srcname, dstname, symlinks, ignore)
            else:
                shutil.copy2(srcname, dstname)
            # XXX What about devices, sockets etc.?
        except (IOError, os.error), why:
            errors.append((srcname, dstname, str(why)))
        # catch the Error from the recursive copytree so that we can
        # continue with other files
        except shutil.Error, err:
            errors.extend(err.args[0])
    try:
        shutil.copystat(src, dst)
    except WindowsError:
        # can't copy file access times on Windows
        pass

Answer 3

This is a wokaround with both distutils and shutil:

ignorePatterns=('.git')
shutil.copytree(src, "__TMP__", ignore=shutil.ignore_patterns(ignorePatterns))
distutils.dir_util.copy_tree("__TMP__", dest)
distutils.dir_util.remove_tree("__TMP__")

Note: This is not an efficient method, because it copies same files twice.