WindowsError: [Error 3] The system cannot find the path specified (when path too long?)

Question

WindowsError: [Error 3] The system cannot find the path specified (when path too long?)

I'm making a script to find unique files between two directories. In order to do this, I use os.walk() to walk through the files, and if files of the same size exist, I hash them to ensure they are the same (opening the files in the process). The problem is that some files produce the above-mentioned error when opened. The most common reason people run into this problem is because the path is not correctly joined, thereby causing the script to try and open a file that doesn't exist. This isn't the case for me.

After trying different combinations of directories, I began to notice a pattern whereby files that produce the error seem to have a deep directory structure and a long filename. I can't think of any other reason for the issue - there are no character-encoding errors (I decode all my paths to UTF-8) and the paths do exist by virtue of os.walk().

My walk code:

for root, dirs, files in os.walk(directory):
    for filename in files:
        file_path = os.path.join(root, filename)

My hashing code:

def hash(file_path):
    with open(dir_file, 'rb') as f:
        hasher = hashlib.md5()
        while True:
            buf = f.read(byte_size)
            if buf != '':
                hasher.update(buf)
            else:
                break
        result = hasher.hexdigest()
    return result

Edit: The most recent path where the issue appeared was 5 directories deep (containing 142 characters, accounting for double backslashes), and the filename was an additional 122 characters long

Answer 1

That's due to Windows API file path size limitation as explained on MSDN:

In the Windows API (with some exceptions discussed in the following paragraphs), the maximum length for a path is MAX_PATH, which is defined as 260 characters. A local path is structured in the following order: drive letter, colon, backslash, name components separated by backslashes, and a terminating null character. For example, the maximum path on drive D is "D:\some 256-character path string" where "" represents the invisible terminating null character for the current system codepage. (The characters < > are used here for visual clarity and cannot be part of a valid path string.)

As also explained on that page, newer versions of Windows support extended file path prefix (\\?\) used for Unicode paths and such, but that's not a consistent or guaranteed behavior i.e. it doesn't mean it will work in all cases.

Either way, try prepending your path with the extended path prefix and see if it works for your case:

file_path = "\\\\?\\" + os.path.join(root, filename)