Why is os.path.expanduser not returning the home directory?

Question

I am making a python desktop application that saves a log as a .csv file in the user's Documents folder on Windows. The application is written in python 2.7 and kivy 1.8.0, packaged as a Windows program using pyinstaller 2.1, and the installer is made using Inno Setup Compiler. In this post, I will replace the user's real name with USER.

I have the following lines of code:

DOCUMENTS = os.path.expanduser('~\\Documents\\')
print DOCUMENTS
with open(DOCUMENTS + 'data_log.csv', 'ab') as f:
    do stuff

On my computer and one other I've tested it on, the program works as expected. DOCUMENTS evaluates to 'C:\Users\USER\Documents\'. However, on the three other computers I've tried, DOCUMENTS evaluates to 'C:\Users\USER\AppData\Roaming\SPB_16.6\Documents\'. The program then crashes when it tries to create data_log.csv, giving the following error:

IOError: [Errno 2] No such file or directory: 'C:\\Users\\USER\\AppData\Roaming\\SPB_16.6\\Documents\\data_log.csv'

First, why might os.path.expanduser misbehave on certain systems, but not on others?

Second, even if it's in the wrong directory, open() should create the file if it doesn't exist, so why is that causing it to crash?

I've figured out what's causing this problem. On most systems, HOME is None, so os.path.expanduser uses USERPROFILE instead. However, in rare cases HOME is set to something like C:\SPB\ or C:\Users\USER\AppData\Roaming\SPB_16.6. My solution is to use os.environ to access USERPROFILE directly instead of using os.path.expanduser.

Answer 1

From the documentation of expanduser:

On Windows, HOME and USERPROFILE will be used if set, otherwise a combination of HOMEPATH and HOMEDRIVE will be used. An initial ~user is handled by stripping the last directory component from the created user path derived above.

As you can see, the code is extremely simple (dumped with inspect):

def expanduser(path):
    """Expand ~ and ~user constructs.

    If user or $HOME is unknown, do nothing."""
    if path[:1] != '~':
        return path
    i, n = 1, len(path)
    while i < n and path[i] not in '/\\':
        i = i + 1

    if 'HOME' in os.environ:
        userhome = os.environ['HOME']
    elif 'USERPROFILE' in os.environ:
        userhome = os.environ['USERPROFILE']
    elif not 'HOMEPATH' in os.environ:
        return path
    else:
        try:
            drive = os.environ['HOMEDRIVE']
        except KeyError:
            drive = ''
        userhome = join(drive, os.environ['HOMEPATH'])

    if i != 1: #~user
        userhome = join(dirname(userhome), path[1:i])

    return userhome + path[i:]

There's not much that can go wrong with expanduser itself. You'll want to check those environment variables inside your program to see if they hold the correct values.

    import os
    for var in ('HOME', 'USERPROFILE', 'HOMEPATH', 'HOMEDRIVE'):
        print os.environ.get(var)

A likely reason open may be failing is that the directory where you're trying to open the file doesn't exist, or you don't have permissions to access it.