Unable to write into DataOutputStream beyond a specific size - OutOfMemoryError

Question

I have the following code which produces an OutOfMemory exception:

byte[] buf = new byte[10240];
int len = 0;
DataOutputStream dataOS = new DataOutputStream(conn.getOutputStream());
while ((len = _inputStream.read(buf)) > 0) {
    System.out.println("len : " + len);
    System.out.println("Going to write buf into dataOS : " + buf.length);
    dataOS.write(buf, 0, len);
    System.out.println("dataOS.size() : " + dataOS.size());
}
_inputStream.close();

The following is the last few lines from the debug output:

len : 10240
Going to write buf into dataOS : 10240
dataOS.size() : 342804702
len : 10240
Going to write buf into dataOS : 10240
dataOS.size() : 342814942
len : 10240
Going to write buf into dataOS : 10240

The exception occurs when I try to write into the dataOS beyond 342814942.

Could someone please help me handle this?

Thanks!

Answer 1

It has nothing to do with the DataOutputStream (which maintains no data whatsoever) and everything to do with your underlying stream conn.getOutputStream(). now, you haven't show the relevant code there, but i'm going to guess "conn" is an instance of HttpURLConnection. i believe the outputstream for HttpURLConnection buffers the output so that it can determine the output length (unless you set it explicitly). if you know the output length, you could set that directly using HttpURLConnection.setFixedLengthStreamingMode, or you could call HttpURLConnection.setChunkedStreamMode to allow the data to be sent in chunks instead of buffered entirely in memory.

in the future, when you encounter an OOME you should always generate a heap dump and open that in a memory profiler. that will almost always show you immediately where the problem is (e.g. in the underlying stream, not the DataOutputStream).

Answer 2

Depending on the implementation of OutputStream, writing may buffer the content in memory, and not actually send it. If you buffer enough content, you will of course run out of memory.

At some interval, perhaps each time through your loop, you should call OutputStream.flush(). If there is a buffer in use, that will clear it.