Python - walk through a huge set of files but in a more efficient manner

Question

I have huge set of files that I want to traverse through using python. I am using os.walk(source) for the same and is working but since I have a huge set of files it is taking too much and memory resources since its getting the complete list all at once. How can I optimize this to use less resources and may be walk through one directory at a time or in some other efficient manner and still able to iterate the complete set of files. Thanks

for dir, dirnames, filenames in os.walk(START_FOLDER): 
    for name in dirnames: 
        #if PRIVATE_FOLDER not in name: 
            for keyword in FOLDER_WITH_KEYWORDS_DELETION_EXCEPTION_LIST: 
                if keyword in name.lower(): 
                    ignoreList.append(name)

Answer 1

If the issue is that the directory simply has too many files in it, this will hopefully be solved in Python 3.5.

Until then, you may want to check out scandir.