Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to unpickle a file that has been hosted in a web URL in python

Tags:

python

pickle

The normal way to pickle and unpickle an object is as follows:

Pickle an object:

import cloudpickle as cp

cp.dump(objects, open("picklefile.pkl", 'wb'))

UnPickle an object: (load the pickled file):

loaded_pickle_object = cp.load(open("picklefile.pkl", 'rb'))

Now, what if the pickled object is hosted in a server, for example a google drive: I am not able to unpickle the object if I directly provide the URL of that object in the path. The following is not working:I get an IOERROR

UnPickle an object: (load the pickled file):

loaded_pickle_object = cp.load(open("https://drive.google.com/file/d/pickled_file", 'rb')) 

Can someone tell me how to load a pickled file into python that is hosted in a web URL?

like image 391
Karthick Mohanraj Avatar asked May 31 '18 12:05

Karthick Mohanraj


People also ask

How do you Unpickle an object in Python?

As we said earlier, the load() method can be used to unpickle the pickled Python object. You have to first open the pickled file using rb (read-binary) permission and pass the opened file to the load() method, as shown below. The load() method unpickles the data and returns the actual object.

Does pickle dump overwrite file?

New! Save questions or answers and organize your favorite content.

What is Unpickle in Python?

“Pickling” is the process whereby a Python object hierarchy is converted into a byte stream, and “unpickling” is the inverse operation, whereby a byte stream (from a binary file or bytes-like object) is converted back into an object hierarchy.


1 Answers

The following has worked for me when importing gdrive pickled files into a Python 3 colab:

from urllib.request import urlopen
loaded_pickle_object = cp.load(urlopen("https://drive.google.com/file/d/pickled_file", 'rb')) 
like image 195
G Pidcock Avatar answered Nov 05 '22 18:11

G Pidcock