Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to show all the element names in a npz file without having to load the completely?

Tags:

numpy

I use the following code to show the element names in a npz file. But it requires loading the file completely, which can be slow especially when the file is large. Is there a way to extract the element names without having to load the file completely?

x = numpy.load(file)
for k in x.iterkeys():
    print k
like image 714
user1424739 Avatar asked Mar 11 '18 11:03

user1424739


1 Answers

Without reading the entire file into memory, you can access small fragments of large files on disk by using mmap [memmap documentation]. Default is r+ (Open existing file for reading and writing).
My test code below uses the NpzFile files attribute [NpzFile documentation], and 'mnist.npz' test data [mnist.npz link], everything seems to be pretty fast in Python 3.6:

>>> import numpy as np
>>> x = np.load('mnist.npz', mmap_mode='r')
>>> for k in x.files:
...     print(k)
... 
x_test
x_train
y_train
y_test
>>> 

Kindly check the linked numpy.memmap for more.

Edit: print(x.files) seems to work fine too.

like image 88
Sam Macharia Avatar answered Nov 21 '22 13:11

Sam Macharia