loop through sub directories, to sample files

Question

The following code selects a random sample of files (in this case 50) from dir 1 and copies them to a new folder with the same name.

However, I have hundreds of folders which I need to sample from (and copy to a new folder with the same name).

How can I adjust the first part of the code so that I can loop through all sub directories, and move the samples to a new folder with the same name. (so the sample of sub dir 1 goes to dir 1, the sample of sub dir 2 goes to dir 2 etc.)

import os 
import shutil 
import random 
from shutil import copyfile

sourcedir = '/home/mrman/dataset-python/train/1/'
newdir  = '/home/mrman/dataset-python/sub-train/1'


filenames = random.sample(os.listdir(sourcedir), 50)
for i in filenames:
    shutil.copy2(sourcedir + i, newdir)

Answer 1

You are looking to use os.walk. Check out the documentation

Run the following to get an understanding of how it works, and read the documentation to understand how this can be used for your solution. Ultimately, what will happen is that you will traverse down the entire directory structure from the path you provide, and each iteration will give you the current path you are at, all the directories in that level, and all the files.

Also, let's say you want to do an operation on a particular full path of something, then make sure you leverage os.path.join when creating your path.

your_path = "/some/path/you/want"
for path, dirs, files in os.walk(your_path):
    print(path)
    print(dirs)
    print(files)