HTTP Headers for Unknown Content-Length

Question

I am currently trying to stream content out to the web after a trans-coding process. This usually works fine by writing binary out to my web stream, but some browsers (specifically IE7, IE8) do not like not having the Content-Length defined in the HTTP header. I believe that "valid" headers are supposed to have this set.

What is the proper way to stream content to the web when you have an unknown Content-Length? The trans-coding process can take awhile, so I want to start streaming it out as it completes.

Answer 1

Try sending them in chunks along with Transfer-Encoding: chunked. More details in wikipedia.

Update as per the comments, here's an example how a "ChunkedOutputStream" in Java may look like:

package com.stackoverflow.q2395192;  import java.io.IOException; import java.io.OutputStream;  public class ChunkedOutputStream extends OutputStream {      private static final byte[] CRLF = "\r\n".getBytes();     private OutputStream output = null;      public ChunkedOutputStream(OutputStream output) {         this.output = output;     }      @Override     public void write(int i) throws IOException {         write(new byte[] { (byte) i }, 0, 1);     }      @Override     public void write(byte[] b, int offset, int length) throws IOException {         writeHeader(length);         output.write(CRLF, 0, CRLF.length);         output.write(b, offset, length);         output.write(CRLF, 0, CRLF.length);     }      @Override     public void flush() throws IOException {         output.flush();     }      @Override     public void close() throws IOException {         writeHeader(0);         output.write(CRLF, 0, CRLF.length);         output.write(CRLF, 0, CRLF.length);         output.close();     }      private void writeHeader(int length) throws IOException {         byte[] header = Integer.toHexString(length).getBytes();         output.write(header, 0, header.length);     }  } 

...which can basically be used as:

OutputStream output = new ChunkedOutputStream(response.getOutputStream()); output.write(....); 

You see in the source, every chunk of data exist of a header which represents the length of data in hex, a CRLF, the actual data and a CRLF. The end of the stream is represented by a header denoting a 0 length and two CRLFs.

Note: despite the example, you actually do not need it in a JSP/Servlet based webapplication. Whenever the content length is not set on a response, the webcontainer will automatically transfer them in chunks.

Answer 2

Just as a follow up to BalusC's excellent post, here is the code I am using in C#. I am chunking data manually directly to an HTTP output stream, after receiving data from the STDOUT on a process.

int buffSize = 16384; byte[] buffer = new byte[buffSize]; byte[] hexBuff; byte[] CRLF = Encoding.UTF8.GetBytes("\r\n");  br = new BinaryReader(transcoder.StandardOutput.BaseStream);  //Begin chunking... int ret = 0; while (!transcoder.HasExited && (ret = br.Read(buffer, 0, buffSize)) > 0) {     //Write hex length...     hexBuff = Encoding.UTF8.GetBytes(ret.ToString("X"));     e.Context.Stream.Write(hexBuff, 0, hexBuff.Length);      //Write CRLF...     e.Context.Stream.Write(CRLF, 0, CRLF.Length);      //Write byte content...     e.Context.Stream.Write(buffer, 0, ret);      //Write CRLF...     e.Context.Stream.Write(CRLF, 0, CRLF.Length); } //End chunking... //Write hex length... hexBuff = Encoding.UTF8.GetBytes(0.ToString("X")); e.Context.Stream.Write(hexBuff, 0, hexBuff.Length);