Cross platform hidden file detection

Question

What is the best way to do cross-platform handling of hidden files? (preferably in Python, but other solutions still appreciated)

Simply checking for a leading '.' works for *nix/Mac, and file attributes work on Windows. However, this seems a little simplistic, and also doesn't account for alternative methods of hiding things (.hidden files, etc.). Is there a standard way to deal with this?

Answer 1

Here's a script that runs on Python 2.5+ and should do what you're looking for:

import ctypes
import os

def is_hidden(filepath):
    name = os.path.basename(os.path.abspath(filepath))
    return name.startswith('.') or has_hidden_attribute(filepath)

def has_hidden_attribute(filepath):
    try:
        attrs = ctypes.windll.kernel32.GetFileAttributesW(unicode(filepath))
        assert attrs != -1
        result = bool(attrs & 2)
    except (AttributeError, AssertionError):
        result = False
    return result

I added something similar to has_hidden_attribute to jaraco.windows. If you have jaraco.windows >= 2.3:

from jaraco.windows import filesystem

def has_hidden_attribute(filepath):
    return filesystem.GetFileAttributes(filepath).hidden

As Ben has pointed out, on Python 3.5, you can use the stdlib:

import os, stat

def has_hidden_attribute(filepath):
    return bool(os.stat(filepath).st_file_attributes & stat.FILE_ATTRIBUTE_HIDDEN)

Though you may still want to use jaraco.windows for the more Pythonic API.

Answer 2

Jason R. Coombs's answer is sufficient for Windows. And most POSIX GUI file managers/open dialogs/etc. probably follow the same "dot-prefix-means-hidden" convention as ls. But not Mac OS X.

There are at least four ways a file or directory can be hidden in Finder, file open panels, etc.:

• Dot prefix.
• HFS+ invisible attribute.
• Finder Info hidden flag.
• Matches a special blacklist built into CoreFoundation (which is different on each OS version—e.g., ~/Library is hidden in 10.7+, but not in 10.6).

Trying to write your own code to handle all of that is not going to be easy. And you'll have to keep it up-to-date, as I'm willing to bet the blacklist will change with most OS versions, Finder Info will eventually go from deprecated to completely unsupported, extended attributes may be supported more broadly than HFS+, …

But if you can require pyobjc (which is already included with recent Apple-supplied Python, and can be installed via pip otherwise), you can just call Apple's code:

import Foundation

def is_hidden(path):
    url = Foundation.NSURL.fileURLWithPath_(path)
    return url.getResourceValue_forKey_error_(None, Foundation.NSURLIsHiddenKey, None)[0]

def listdir_skipping_hidden(path):
    url = Foundation.NSURL.fileURLWithPath_(path)
    fm = Foundation.NSFileManager.defaultManager()
    urls = fm.contentsOfDirectoryAtURL_includingPropertiesForKeys_options_error_(
        url, [], Foundation.NSDirectoryEnumerationSkipsHiddenFiles, None)[0]
    return [u.path() for u in urls]

This should work on any Python that pyobjc supports, on OS X 10.6+. If you want 10.5 or earlier, directory enumeration flags didn't exist yet, so the only option is something like filtering something like contentsOfDirectoryAtPath_error_ (or just os.listdir) on is_hidden.

If you have to get by without pyobjc, you can drop down to the CoreFoundation equivalents, and use ctypes. The key functions are CFURLCopyResourcePropertyForKey for is_hidden and CFURLEnumeratorCreateForDirectoryURL for listing a directory.

See http://pastebin.com/aCUwTumB for an implementation.

I've tested with:

• OS X 10.6, 32-bit python.org 3.3.0
• OS X 10.8, 32-bit Apple 2.7.2
• OS X 10.8, 64-bit Apple 2.7.2
• OS X 10.8, 64-bit python.org 3.3.0

It works as appropriate on each (e.g., it skips ~/Library on 10.8, but shows it on 10.6).

It should work on any OS X 10.6+ and any Python 2.6+. If you need OS X 10.5, you need to use the old APIs (or os.listdir) and filter on is_hidden. If you need Python 2.5, change the bytes checks to str checks (which of course breaks 3.x) and the with to an ugly try/finally or manual releasing.

If anyone plans on putting this code into a library, I would strongly suggest checking for pyobjc first (import Foundation and, if you don't get an ImportError you win), and only using the ctypes code if it's not available.

---

One last note:

Some people looking for this answer are trying to reinvent a wheel they don't need to.

Often, when people are doing something like this, they're building a GUI and want to, e.g., show a file browsers with an option to hide or show hidden files. Many of the popular cross-platform GUI frameworks (Qt, wx, etc.) have this support built in. (Also, many of them are open source, so you can read their code to see how they do it.)

That may not answer your question—e.g., they may just be passing a "filter hidden files" flag to the platform's native file-browser dialog, but you're trying to build a console-mode file-browser and can't do that. But if it does, just use it.

Answer 3

We actually address this in a project we write. What we do is have a number of different "hidden file checkers" that are registered with a main checker. We pass each file through these to see if it should be hidden or not.

These checkers are not only for different OS's etc, but we plug into version control "ignored" files, and optional user overrides by glob or regular expression.

It mostly amounts to what you have done, but in a pluggable, flexible and extensible way.

See source code here: https://bitbucket.org/aafshar/pida-main/src/tip/pida/services/filemanager/filemanager.py