so when I use the savemat command it tends to overwrite the file. Is there a possible way to append instead of overwriting? I know a work-around would be to put everything into a list, and then convert it to a dictionary. That won't work for me because I am trying to be RAM efficient. Doing online search I found this How NOT to overwrite the .mat file when using scipy.io.savemat()?
that won't work either because it involves pulling a data file in your ram memory to append it every single loop, which seems stupid from the speed perspective.
I thought about appending to a numpy binary file, then pulling that in and saving it to a .mat file? I am not sure if this would be more RAM efficient than the first option though.
thanks!
According to savemat
docs:
file_name : str or file-like object
So you can open the file in append mode, and write, e.g.
io.savemat('temp.mat',{'data':np.ones(10)}) # write
with open('temp.mat','ab') as f:
io.savemat(f, {'newdata':np.arange(5)}) # append
print io.loadmat('temp.mat').keys() # read
# dict_keys(['data', '__globals__', 'newdata', '__header__', '__version__'])
Or the full description:
{'data': array([[ 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]]),
'__globals__': [],
'newdata': array([[0, 1, 2, 3, 4]]),
'__header__': b'MATLAB 5.0 MAT-file Platform: posix, Created on: Fri Mar 13 14:14:33 2015',
'__version__': '1.0'}
A note in https://github.com/scipy/scipy/blob/master/scipy/io/matlab/mio5.py#L34 suggests that there is a problem with appending when there's a function in the data file, but this indicates that appending isn't a problem if we are just saving arrays. But maybe further search of the scipy
issues is in order.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With