How to get progress of os.walk in python?

Question

I have a piece of code which I'm using to search for the executables of game files and returning the directories. I would really like to get some sort of progress indicator as to how far along os.walk is. How would I accomplish such a thing?

I tried doing startpt = root.count(os.sep) and gauging off of that but that just gives how deep os.walk is in a directory tree.

def locate(filelist, root=os.curdir): #Find a list of files, return directories.
    for path, dirs, files in os.walk(os.path.abspath(root)):
        for filename in returnMatches(filelist, [k.lower() for k in files]):
            yield path + "\\"

Answer 1

It depends!

If the files and directories are distributed more or less evenly you could show rough process by assuming every toplevel directory is going to take the same amount of time. But if they are not distributed evenly you cannot find out about it cheaply. You either have to know roughly how populated every directory is in advance, or you have to os.walk the entire thing twice (but that is only useful if your actual processing takes much longer than the os.walk itself does).

That is: say you have 4 toplevel directories, and each one contains 4 files. If you assume every toplevel dir takes 25% of progress, and each file takes another 25% of the progress for that dir, you can show a nice progress indicator. But if the last subdir turns out to contain many more files than the first few your progress indicator will have hit 75% before you find out about it. You cannot really fix that if the os.walk itself is the bottleneck (not your processing) and it's an arbitrary directory tree (not one where you know in advance roughly how long every subtree is going to take).

And of course that's assuming the cost here is about the same for every file...

Answer 2

Just show an indeterminate progress bar (i.e. the ones that show a blob bouncing back and forth or the barber pole effect). That way users know that the program is doing something useful but doesn't mislead them as far as time to complete and such.