Python can't open symlinked file

Question

Python is unable to open my simlinked file. I made sure the file exists and I can access it. I was under the impression that symlinks are resolved on the OS level so Python would never know about it.

therold@therold:~/Programming/ML/deeptank/dataset/inc (master)$ ls -lisa  /Users/therold/Programming/ML/deeptank/dataset/inc/training/tanks/matilda-iv_689.png
7870541 8 lrwxr-xr-x  1 therold  staff  46 13 Mai 16:44 /Users/therold/Programming/ML/deeptank/dataset/inc/training/tanks/matilda-iv_689.png -> ../training_data/matilda-iv/matilda-iv_689.png

OSX is clearly able resolve the symlinked image file. However the Python open() method fails me:

therold@therold:~/Programming/ML/deeptank/dataset/inc (master)$ python
Python 2.7.10 (default, Sep 23 2015, 04:34:21)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.72)] on darwin
>>> open("/Users/therold/Programming/ML/deeptank/dataset/inc/training/tanks/matilda-iv_689.png")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IOError: [Errno 2] No such file or directory: '/Users/therold/Programming/ML/deeptank/dataset/inc/training/tanks/matilda-iv_689.png'
>>>

Any ideas what I am doing wrong? Help is much appreciated.

Answer 1

Any ideas what I am doing wrong?

The target of the symbolic link doesn't exist.

I don't understand why I was able to resolve it in the ls statement in my question.

You weren't.

The ls command by default operates on the link itself, not on the target of the link. Absent the -L option, ls never attempts to resolve the symbolic link.

Consider a directory with these two files:

$ ls -l
-rw-rw-r-- 1 me me 6 May 13 11:58 a
lrwxrwxrwx 1 me me 3 May 13 12:00 b -> ./a

a is a text file containing the six bytes 'hello\n'. b is a link file containing the three bytes to its target path: './a'. ls is able to describe the properties of the link without dereferencing the link itself.

In contrast, use the -L option:

$ ls -lL
-rw-rw-r-- 1 me me 6 May 13 11:58 a
-rw-rw-r-- 1 me me 6 May 13 11:58 b

Now ls has resolved the link in b, and displays information about the linked-to file. With -L, ls now claims that b is also a six-byte text file.

Finally, consider this:

$ rm a
$ ls -l
lrwxrwxrwx 1 me me 3 May 13 12:00 b -> ./a
$ ls -lL
ls: cannot access b: No such file or directory
l????????? ? ? ? ?            ? b

Now b is a link that resolves to a file that no longer exists. Since ls -l never attempts the resolve the link, its output is unchanged from the previous test. (b is a link file, 3 bytes long, contents './a'.)

But ls -lL attempts to resolve the link, fails, and displays the failure information.

Answer 2

... don't forget to use complete paths. On Raspberry Pi with Python 3.7.3 I had to include the whole paths for a working link:

os.symlink("Desktop/a.txt","Desktop/a_link.txt")  # WRONG !

python3 did not recognize the file a.txt because the paths of the standard shell are not active.

os.symllink("/home/pi/Desktop/a.txt","/home/pi/Desktop/a_link.txt")

works well.

Answer 3

I'm logging an answer because I came to this question with the same problem, but the target of my symlink did exist.

Trying to open a file across a symlink in pandas gave an unhelpful not-found message:

⇒  ipython.exe  # this is a clue to the answer below...

In [1]: import pandas as pd

In [2]: codes = pd.read_excel('codes-no-agg-children.xlsx', ...)
---------------------------------------------------------------------------
<...lots of traceback stuff...>
FileNotFoundError: [Errno 2] No such file or directory: 'codes-no-agg-children.xlsx'   

Since the target did exist, the answer above didn't help me, but the question prompted me to try a regular python open instead of through pandas:

In [3]: import os                                                                                                                                                                                                                                         
In [4]: os.system('cp ./nvivo-win/good-extracts/codes-no-agg-children.xlsx .')                                               
\\wsl$\Ubuntu\home\sigfried\ccvs\analysis'                                                                                  
CMD.EXE was started with the above path as the current directory.                                                            
UNC paths are not supported.  Defaulting to windows directory.                                                               
cp: cannot stat './nvivo-win/good-extracts/codes-no-agg-children.xlsx': No such file or directory

I don't know what all that means, but I assume the cause is:

Python for windows can't follow Ubuntu (WSL 2) symlinks (or, at least it can't open Ubuntu symlinks back to a windows file.

Hope this helps someone.

Answer 4

I had the same issue-Fedora 32 system, with Jupyter's Notebook folder containing my files having symlinks to json files. I beat my head against the proverbial wall for a few days, googling came up short. In the end, copying the files over fixed the issue (though there are other options like hardlinks).