I can't load my model because I can't put a PosixPath

Question

I'm setting up a script and I need to use some functions from fast-ai package. The fact is that I'm on Windows and when I define my paths, the function from fast-ai named load_learner can't load the model.

I've tried to change the function into the package as:

state = pickle.load(open(str(path) + '/' + str(fname), 'rb'))

instead of:

state = pickle.load(open(path/fname, 'rb'))

but I obtain this error:

 File "lib\site-packages\fastai\basic_train.py", line 462, in load_learner
    state = pickle.load(open(path/fname, 'rb'))
  File "\lib\pathlib.py", line 1006, in __new__
    % (cls.__name__,))
NotImplementedError: cannot instantiate 'PosixPath' on your system

My paths are defined as:

folder_path = './models/model1'
fname = 'model.pkl'

and I call the function as: model = load_learner(folder_path, fname)

How can I use Windows paths in this function?

---

UPDATE 1

The answer posted was correct only on Linux. I still have the issue on Windows. I didn't find a way to pass through the PosixPath on Windows. The only solution that I found is to change the internal packages from my modules but it is not a secure way to solve this kind of issue.

---

Thanks in advance.

Answer 1

Just redirect PosixPath to WindowsPath.

import pathlib
temp = pathlib.PosixPath
pathlib.PosixPath = pathlib.WindowsPath

I am also loading fastai models and this trick works.

Answer 2

According my own question, I find a way using:

from pathlib import Path

folder_path = Path('./models/model1')

---

UPDATE 1

This solution only works on Linux, on Windows I still get an error.

---

Answer 3

According to provided error message you are using pathlib. So you don't need to use + '/' + here: str(path) + '/' + str(fname)

/ as path separator works on Linux/Unix:

state = pickle.load(open(path / fname, 'rb'))

On Windows use .joinpath() instead:

state = pickle.load(open(path.joinpath(fname), 'rb'))

If you are not going to use the pathlib, use os.path.join(). It will automatically select right format for your OS.

Answer 4

The issue here is related to the differences in the way Python handles paths depending on the OS:

• PosixPath - on Linux / Unix

• WindowsPath - on Windows

When persisting objects using pickle on one OS (say Linux - as in this case) information about the type / class is persisted too (here: PosixPath).

Now, when the pickle file is being loaded Python assumes it is going to be able to recreate objects based on the type information it previously persisted. In this case it tries to recreate an object of PosixPath type which is prevented by pathlib library and cannot be instantiated on Windows. On Windows there should be WindowsPath be used instead but pickle module does not handle such OS-dependent logic very well hence it helplessly throws the error.

You could theoretically interfere in the code of pathlib to remove the OS check but there is no easy workaround but avoiding pickling of OS-dependent objects (e.g. storing paths as strings - as os.path does - would certainly workaround this issue).

There is also another possibility - to use a platform-independent PurePosixPath class for path objects.