Why path is not found in python?

Question

I am trying to open a file that exists using python, and opens perfectly if I open it using gedit in command line.

However, I get the following error message:

andreas@ubuntu:~/Desktop/Thesis/Codes/ModifiedFiles$ python vis.py -f myoutputcsv.csv
Matplotlib version 1.3.1
Traceback (most recent call last):
  File "vis.py", line 1082, in <module>
    reliability_table = ReliabilityTable(reliability_table_file)
  File "vis.py", line 112, in __init__
    self.read(filename)
  File "vis.py", line 139, in read
    self.data = genfromtxt(filename, delimiter=',',comments='#', dtype=float)
  File "/usr/lib/python2.7/dist-packages/numpy/lib/npyio.py", line 1344, in genfromtxt
    fhd = iter(np.lib._datasource.open(fname, 'rbU'))
  File "/usr/lib/python2.7/dist-packages/numpy/lib/_datasource.py", line 147, in open
    return ds.open(path, mode)
  File "/usr/lib/python2.7/dist-packages/numpy/lib/_datasource.py", line 496, in open
    raise IOError("%s not found." % path)
IOError: ~/Desktop/Thesis/Codes/ModifiedFiles/reliability_table_2.csv not found.

Do you know what I may be doing wrong? I have very little experience with python and I cannot find the reason the file opens at command line but not using python.

Answer 1

The ~ (tilde) is a shell expansion, not a special "filesystem expansion". So ~ expands to the current user directly only when found in a shell command:

$echo ~
/home/username

But not if used in the filename passed to python's file objects. The python code:

open('some/file/name')

is equivalent to opening the file 'some/file/name' in the shell, and I mean literally with the single quotes that prevent expansions included.

So:

open('~/file.txt')

Tries to open:

$echo '~/file.txt'
~/file.txt

And not:

$echo ~/file.txt
/home/username/file.txt

This is stated at the top of the documentation of the os.path module too:

Unlike a unix shell, Python does not do any automatic path expansions. Functions such as expanduser() and expandvars() can be invoked explicitly when an application desires shell-like path expansion. (See also the glob module.)

---

In fact you can create a file called ~:

$touch '~'
$ls | grep '~'
~

The single quotes are necessary, because touch ~ would simply execute touch on /home/username and no file would be created.

Now if you try to delete it you must escape its name, otherwise the shell will expand it into /home/username:

$echo ~
/home/username
$rm ~     # translation: "rm: cannot remove "/home/username": It's a directory"
rm: impossibile rimuovere "/home/username": È una directory
$rm '~'   # proper way to delete it

---

If you want to expand the ~ in filenames use the os.path.expanduser function:

>>> import os.path
>>> os.path.expanduser('~/file.txt')
'/home/username/file.txt'

Note that realpath and abspath do not expand the ~:

>>> os.path.realpath('~/file.txt')
'/home/username/~/file.txt'
>>> os.path.abspath('~/file.txt')
'/home/username/~/file.txt'

So, if you want to be sure to convert a pathname given by the user in "shell language"¹ into an absolute path usable with python's file objects you should do:

os.path.abspath(os.path.expanduser(path))

---

¹ Not saying sh/bash because they are cross platform.