shutil moving files keeping the same directory structure

Question

I want to move a lot of files. the path to these files is stored in a list. I want to keep the whole directory structure but want to move them to a different folder.

So for example the files are D:\test\test1\test1.txt D:\test\test1\test2.txt

I want to move them to C:\ from D:\ and keep the directory structure. How should I go about doing it?

this is the code i have, it is not working

import os, fnmatch
import shutil


f=open('test_logs.txt','r') #logs where filenames are stored with filenames as first entry

for line in f:
    filename=line.split()
    output_file="C:" + filename[0].lstrip("D:")
    shutil.move(filename[0],output_file)

I read the file name fine and I can generate the destination filename fine but when I run it, it gives me an error saying "No such file or directory" (and gives the path of the output filename).

Answer 1

I think you want something like this:

import sys
import os
import shutil

# terminology:
# path = full path to a file, i.e. directory + file name
# directory = directory, possibly starting with a drive
# file name = the last component of the path

sourcedrive = 'D:'
destdrive = 'C:'

log_list_file = open('test_logs.txt', 'r')
for line in log_list_file:
    sourcepath = line.split()[0]  # XXX is this correct?
    if sourcepath.startswith(sourcedrive):
        destpath = sourcepath.replace(sourcedrive, destdrive, 1)
    else:
        print >>sys.stderr, 'Skipping %s: Not on %s' % (sourcepath, sourcedrive)
        continue

    destdir = os.path.dirname(destpath)    

    if not os.path.isdir(destdir):
        try:
            os.makedirs(destdir)
        except (OSError, IOError, Error) as e:
            print >>sys.stderr, 'Error making %s: %s' % (destdir, e)
            continue

    try:
        shutil.move(sourcepath, destpath)
    except (OSError, IOError, Error) as e:
        print >>sys.stderr, 'Error moving %s to %s: %s' % (sourcepath, destpath, e)

Do you also want to remove the source directory if it's empty?

Answer 2

Update: Ah, ok, I see the problem -- shutil.move won't copy to a nonexistent directory. To do what you're trying to do, you have to create the new directory tree first. Since it's a bit safer to use a built-in move function than to roll your own copy-and-delete procedure, you could do this:

with open('test_logs.txt','r') as f:
    files_to_copy = [line.split()[0] for line in f]
paths_to_copy = set(os.path.split(filename)[0] for filename in files_to_copy)

def ignore_files(path, names, ptc=paths_to_copy):
    return [name for name in names if os.path.join(path, name) not in ptc]

shutil.copytree(src, dst, ignore=ignore_files)

for filename in files_to_copy:
    output_file="C:" + filename.lstrip("D:")
    shutil.move(filename, output_file)

Let me know if that doesn't work

---

Original Post: If you want to move only some of the files, your best bet is to use shutil.copytree's ignore keyword. Assuming your list of files includes full paths and directories (i.e. ['D:\test\test1\test1.txt', 'D:\test\test1\test2.txt', 'D:\test\test1']), create an ignore_files function and use it like this:

files_to_copy = ['D:\test\test1\test1.txt', 'D:\test\test1\test2.txt', 'D:\test\test1']

def ignore_files(path, names, ftc=files_to_copy):
    return [name for name in names if os.path.join(path, name) not in ftc]

shutil.copytree(src, dst, ignore=ignore_files)

Then you can just delete the files in files_to_copy:

for f in files_to_copy:
    try:
        os.remove(f)
    except OSError:  # can't remove() a directory, so pass
        pass 

I tested this -- make certain that you include the paths you want to copy as well as the files in files_to_copy -- otherwise, this will delete files without copying them.