Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Finding shape of saved numpy array (.npy or .npz) without loading into memory

Tags:

python

io

numpy

I have a huge compressed numpy array saved to disk (~20gb in memory, much less when compressed). I need to know the shape of this array, but I do not have the available memory to load it. How can I find the shape of the numpy array without loading it into memory?

like image 234
pir Avatar asked Mar 14 '16 14:03

pir


2 Answers

This does it:

import numpy as np
import zipfile

def npz_headers(npz):
    """Takes a path to an .npz file, which is a Zip archive of .npy files.
    Generates a sequence of (name, shape, np.dtype).
    """
    with zipfile.ZipFile(npz) as archive:
        for name in archive.namelist():
            if not name.endswith('.npy'):
                continue

            npy = archive.open(name)
            version = np.lib.format.read_magic(npy)
            shape, fortran, dtype = np.lib.format._read_array_header(npy, version)
            yield name[:-4], shape, dtype
like image 200
John Zwinck Avatar answered Nov 04 '22 22:11

John Zwinck


Opening the file in mmap_mode might do the trick.

    If not None, then memory-map the file, using the given mode
    (see `numpy.memmap` for a detailed description of the modes).
    A memory-mapped array is kept on disk. However, it can be accessed
    and sliced like any ndarray.  Memory mapping is especially useful for
    accessing small fragments of large files without reading the entire
    file into memory.

It is also possible to read the header block without reading the data buffer, but that requires digging further into the underlying lib/npyio/format code. I explored that in a recent SO question about storing multiple arrays in a single file (and reading them).

https://stackoverflow.com/a/35752728/901925

like image 44
hpaulj Avatar answered Nov 04 '22 20:11

hpaulj