Flushing numpy memmap to npy file

Question

Is there a method to save a numpy memmap array into a .npy file? Apparently, there is a method to load such an array from a .npy file as follows

data = numpy.load("input.npy", mmap_mode='r')

but flushing the file is not equivalent to storing it in a .npy format.

If flushing is the only way to go then is there a way to infer the shape of the stored array? I would prefer to have dynamic shape which is automatically stored and retrieved (possibly as memmap again) in another script.

I have searched on various places about this but didn't find get any result. I way to store into .npy I do now is

numpy.save(output.filename, output.copy())

which defeats the idea of using memmap but preserves the shape.

NOTE: I know about hdf5 and h5py but I was wondering if there is a pure numpy solution to this.

Answer 1

is there a way to infer the shape of the stored array?

No. As far as np.memmap is concerned the file is just a buffer - it stores the contents of the array, but not the dimensions, dtype etc. There's no way to infer that information unless it's somehow contained within the array itself. If you've already created an np.memmap backed by a simple binary file then you would need to write its contents to a new .npy file on disk.

You could avoid generating a copy in memory by opening the new .npy file as another memory-mapped array using numpy.lib.format.open_memmap:

import numpy as np
from numpy.lib.format import open_memmap

# a 10GB memory-mapped array
x = np.memmap('/tmp/x.mm', mode='w+', dtype=np.ubyte, shape=(int(1E10),))

# create a memory-mapped .npy file with the same dimensions and dtype
y = open_memmap('/tmp/y.npy', mode='w+', dtype=x.dtype, shape=x.shape)

# copy the array contents
y[:] = x[:]