Read/Write Bytes to and From a File Using Only Java.IO

Question

How can we write a byte array to a file (and read it back from that file) in Java?

Yes, we all know there are already lots of questions like that, but they get very messy and subjective due to the fact that there are so many ways to accomplish this task.

So let's reduce the scope of the question:

Domain:

• Android / Java

What we want:

• Fast (as possible)
• Bug-free (in a rigidly meticulous way)

What we are not doing:

• Third-party libraries
• Any libraries that require Android API later than 23 (Marshmallow)

(So, that rules out Apache Commons, Google Guava, Java.nio, and leaves us with good ol' Java.io)

What we need:

• Byte array is always exactly the same (content and size) after going through the write-then-read process
• Write method only requires two arguments: File file, and byte[] data
• Read method returns a byte[] and only requires one argument: File file

In my particular case, these methods are private (not a library) and are NOT responsible for the following, (but if you want to create a more universal solution that applies to a wider audience, go for it):

• Thread-safety (file will not be accessed by more than one process at once)
• File being null
• File pointing to non-existent location
• Lack of permissions at the file location
• Byte array being too large
• Byte array being null
• Dealing with any "index," "length," or "append" arguments/capabilities

So... we're sort of in search of the definitive bullet-proof code that people in the future can assume is safe to use because your answer has lots of up-votes and there are no comments that say, "That might crash if..."

This is what I have so far:

Write Bytes To File:

private void writeBytesToFile(final File file, final byte[] data) {
        try {
            FileOutputStream fos = new FileOutputStream(file);
            fos.write(data);
            fos.close();
        } catch (Exception e) {
            Log.i("XXX", "BUG: " + e);
        }
    }

Read Bytes From File:

private byte[] readBytesFromFile(final File file) {
        RandomAccessFile raf;
        byte[] bytesToReturn = new byte[(int) file.length()];
        try {
            raf = new RandomAccessFile(file, "r");
            raf.readFully(bytesToReturn);
        } catch (Exception e) {
            Log.i("XXX", "BUG: " + e);
        }
        return bytesToReturn;
}

From what I've read, the possible Exceptions are:

FileNotFoundException : Am I correct that this should not happen as long as the file path being supplied was derived using Android's own internal tools and/or if the app was tested properly?

IOException : I don't really know what could cause this... but I'm assuming that there's no way around it if it does.

So with that in mind... can these methods be improved or replaced, and if so, with what?

Answer 1

It looks like these are going to be core utility/library methods which must run on Android API 23 or later.

Concerning library methods, I find it best to make no assumptions on how applications will use these methods. In some cases the applications may want to receive checked IOExceptions (because data from a file must exist for the application to work), in other cases the applications may not even care if data is not available (because data from a file is only cache that is also available from a primary source).

When it comes to I/O operations, there is never a guarantee that operations will succeed (e.g. user dropping phone in the toilet). The library should reflect that and give the application a choice on how to handle errors.

To optimize I/O performance always assume the "happy path" and catch errors to figure out what went wrong. This is counter intuitive to normal programming but essential in dealing with storage I/O. For example, just checking if a file exists before reading from a file can make your application twice as slow - all these kind of I/O actions add up fast to slow your application down. Just assume the file exists and if you get an error, only then check if the file exists.

So given those ideas, the main functions could look like:

public static void writeFile(File f, byte[] data) throws FileNotFoundException, IOException {
    try (FileOutputStream out = new FileOutputStream(f)) {
        out.write(data);
    }
}

public static int readFile(File f, byte[] data) throws FileNotFoundException, IOException {
    try (FileInputStream in = new FileInputStream(f)) {
        return in.read(data); 
    }
}

Notes about the implementation:

• The methods can also throw runtime-exceptions like NullPointerExceptions - these methods are never going to be "bug free".
• I do not think buffering is needed/wanted in the methods above since only one native call is done (see also here).
• The application now also has the option to read only the beginning of a file.

To make it easier for an application to read a file, an additional method can be added. But note that it is up to the library to detect any errors and report them to the application since the application itself can no longer detect those errors.

public static byte[] readFile(File f) throws FileNotFoundException, IOException {
    int fsize = verifyFileSize(f);
    byte[] data = new byte[fsize];
    int read = readFile(f, data);
    verifyAllDataRead(f, data, read);
    return data;
}

private static int verifyFileSize(File f) throws IOException {
    long fsize = f.length();
    if (fsize > Integer.MAX_VALUE) {
        throw new IOException("File size (" + fsize + " bytes) for " + f.getName() + " too large.");
    }
    return (int) fsize;
}

public static void verifyAllDataRead(File f, byte[] data, int read) throws IOException {
    if (read != data.length) {
        throw new IOException("Expected to read " + data.length 
                + " bytes from file " + f.getName() + " but got only " + read + " bytes from file.");
    }
}

This implementation adds another hidden point of failure: OutOfMemory at the point where the new data array is created.

To accommodate applications further, additional methods can be added to help with different scenario's. For example, let's say the application really does not want to deal with checked exceptions:

public static void writeFileData(File f, byte[] data) {
    try {
        writeFile(f, data);
    } catch (Exception e) {
        fileExceptionToRuntime(e);
    }
}

public static byte[] readFileData(File f) {
    try {
        return readFile(f);
    } catch (Exception e) {
        fileExceptionToRuntime(e);
    }
    return null;
}

public static int readFileData(File f, byte[] data) {
    try {
        return readFile(f, data);
    } catch (Exception e) {
        fileExceptionToRuntime(e);
    }
    return -1;
}

private static void fileExceptionToRuntime(Exception e) {
    if (e instanceof RuntimeException) { // e.g. NullPointerException
        throw (RuntimeException)e;
    }
    RuntimeException re = new RuntimeException(e.toString());
    re.setStackTrace(e.getStackTrace());
    throw re;
}

The method fileExceptionToRuntime is a minimal implementation, but it shows the idea here.

The library could also help an application to troubleshoot when an error does occur. For example, a method canReadFile(File f) could check if a file exists and is readable and is not too large. The application could call such a function after a file-read fails and check for common reasons why a file cannot be read. The same can be done for writing to a file.

Answer 2

Although you can't use third party libraries, you can still read their code and learn from their experience. In Google Guava for example, you usually read a file into bytes like this:

FileInputStream reader = new FileInputStream("test.txt");
byte[] result = ByteStreams.toByteArray(reader);

The core implementation of this is toByteArrayInternal. Before calling this, you should check:

• A not null file is passed (NullPointerException)
• The file exists (FileNotFoundException)

After that, it is reduced to handling an InputStream and this where IOExceptions come from. When reading streams a lot of things out of the control of your application can go wrong (bad sectors and other hardware issues, mal-functioning drivers, OS access rights) and manifest themselves with an IOException.

I am copying here the implementation:

private static final int BUFFER_SIZE = 8192;

/** Max array length on JVM. */
private static final int MAX_ARRAY_LEN = Integer.MAX_VALUE - 8;

private static byte[] toByteArrayInternal(InputStream in, Queue<byte[]> bufs, int totalLen)
      throws IOException {
    // Starting with an 8k buffer, double the size of each successive buffer. Buffers are retained
    // in a deque so that there's no copying between buffers while reading and so all of the bytes
    // in each new allocated buffer are available for reading from the stream.
    for (int bufSize = BUFFER_SIZE;
        totalLen < MAX_ARRAY_LEN;
        bufSize = IntMath.saturatedMultiply(bufSize, 2)) {
      byte[] buf = new byte[Math.min(bufSize, MAX_ARRAY_LEN - totalLen)];
      bufs.add(buf);
      int off = 0;
      while (off < buf.length) {
        // always OK to fill buf; its size plus the rest of bufs is never more than MAX_ARRAY_LEN
        int r = in.read(buf, off, buf.length - off);
        if (r == -1) {
          return combineBuffers(bufs, totalLen);
        }
        off += r;
        totalLen += r;
      }
    }

    // read MAX_ARRAY_LEN bytes without seeing end of stream
    if (in.read() == -1) {
      // oh, there's the end of the stream
      return combineBuffers(bufs, MAX_ARRAY_LEN);
    } else {
      throw new OutOfMemoryError("input is too large to fit in a byte array");
    }
  }

As you can see most of the logic has to do with reading the file in chunks. This is to handle situations, where you don't know the size of the InputStream, before starting reading. In your case, you only need to read files and you should be able to know the length beforehand, so this complexity could be avoided.

The other check is OutOfMemoryException. In standard Java the limit is too big, however in Android, it will be a much smaller value. You should check, before trying to read the file that there is enough memory available.