Saving large short array android

Question

I'm basically looking for a way to save and reload a few large arrays (about 5 million short) in a fastish way on Android. My app needs to save them in a way where I could get back to them much later, so I can't just hold them in memory...

So far, I've tried converting them to a byte[] array and they seem to save successfully, but I can't get the data back, this is how my saving code works (it's actually to separate functions, simplified here):

ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOuputStream oos = new ObjectOutputStream(baos);
oos.writeObject(data);
oos.close();
baos.close();
FileOutputStream fos = new FileOutputStream(filename); // valid absolute path
fos.write(baos.toByteArray());
fos.close();

And the loading part is where I get stuck, how do I get a short[] from a byte[]? I also read that a database might work too, but is it quick enough?

I've looked all around Stackoverflow and Google and can't seem to find somebody with a similar problem, or at least a solution to it, but as a beginner I might have missed something obvious...

Answer 1

If data is a short[] and you want to write the whole thing into a file, don't use a buffer in memory, write it directly.

ObjectOutputStream oos = new ObjectOutputStream(new FileOutputStream(filename));
try {
    oos.writeObject(data);
} finally {
    oos.close();
}

If you write it via ObjectOutputStream, you will have to read it via ObjectInputStream since it's not just writing the pure data but also some type information. If you put a short[] in, you get a short[] back (you could try to skip those bytes but you would have to analyze what the stream is actually writing). This also applies if your ObjectOutputStream writes into a ByteArrayOutputStream.

If you don't want to handle that mess, do what ObjectOutputStream basically does:

DataOutputStream dos = new DataOutputStream(new FileOutputStream(filename));
try {
    for (int i = 0; i < data.length; i++) {
        dos.writeShort(data[i]);
    }
} finally {
    dos.close();
}

DataOutputStream writes plain data so you can simply read the data directly as a byte[] if you want. If byte order matters: it's using Big-Endian.

---

Since this approach writes single bytes instead of chucks of byte[] it benefits from using a BufferedOutputStream in between. The default uses an 8kB buffer, but it can increased which could give even better results. Writing the data will now convert short into bytes in memory and once enough data is available the whole chunk will get pushed to low level OS functions to write it.

int bufferSize = 32 * 1024;
DataOutputStream dos = new DataOutputStream(
        new BufferedOutputStream(
                new FileOutputStream(filename),
                bufferSize)
        );
try {
    for (int i = 0; i < data.length; i++) {
        dos.writeShort(data[i]);
    }
} finally {
    dos.close();
}

Answer 2

Use the file streams directly without buffering through an ByteArrayOutputStream / ByteArrayInputStream. The byte[] is held in memory and not suitable for large potions of data.

Writing to an new ObjectOutputStream(new FileOutputStream(filename)), you can get your objects back using an new ObjectInputStream(new FileInputStream(filename)). If you want, you can also do some buffering as mentioned in Christopher's answer.