Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Chunked transfer encoding - browser behavior

People also ask

How does Transfer-Encoding chunked work?

In chunked transfer encoding, the data stream is divided into a series of non-overlapping "chunks". The chunks are sent out and received independently of one another. No knowledge of the data stream outside the currently-being-processed chunk is necessary for both the sender and the receiver at any given time.

What is meant by HTTP chunking?

Chunking is a technique that HTTP servers use to improve responsiveness. Chunking can help you avoid situations where the server needs to obtain dynamic content from an external source and delays sending the response to the client until receiving all of the content so the server can calculate a Content-Length header.

How do I turn off chunked transfer encoding?

Try adding "&headers=false" to your request. That should shorten it up and cause the response to be less likely to be chunked. Also, are you sending a HTTP/1.1 or HTTP/1.0 request? Try sending a HTTP/1.0 if your device cannot handle a HTTP/1.1 request.


afaik browsers needs some payload to start render chunks as they received.
Curl is of course an exception.

Try to send about 1KB of arbitrary data before your first chunk.

If you are doing everything correctly, browsers should render chunks as they received.


Fix your headers.


  1. As of 2019, if you use Content-type: text/html, no buffering occurs in Chrome.

  1. If you just want to stream text, similar to text/plain, then just using Content-type: text/event-stream will also disable buffering.

  1. If you use Content-type: text/plain, then Chrome will still buffer 1 KiB, unless you additionally specify X-Content-Type-Options: nosniff.

RFC 2045 specifies that if no Content-Type is specified, Content-type: text/plain; charset=us-ascii should be assumed

5.2. Content-Type Defaults

Default RFC 822 messages without a MIME Content-Type header are taken by this protocol to be plain text in the US-ASCII character set, which can be explicitly specified as:

Content-type: text/plain; charset=us-ascii

This default is assumed if no Content-Type header field is specified. It is also recommend that this default be assumed when a syntactically invalid Content-Type header field is encountered. In the presence of a MIME-Version header field and the absence of any Content-Type header field, a receiving User Agent can also assume that plain US-ASCII text was the sender's intent. Plain US-ASCII text may still be assumed in the absence of a MIME-Version or the presence of an syntactically invalid Content-Type header field, but the sender's intent might have been otherwise.

Browsers will start to buffer text/plain for a certain amount in order to check if they can detect if the content sent is really plain text or some media type like an image, in case the Content-Type was omitted, which would then equal a text/plain content type. This is called MIME type sniffing.

MIME type sniffing is defined by Mozilla as:

In the absence of a MIME type, or in certain cases where browsers believe they are incorrect, browsers may perform MIME sniffing — guessing the correct MIME type by looking at the bytes of the resource.

Each browser performs MIME sniffing differently and under different circumstances. (For example, Safari will look at the file extension in the URL if the sent MIME type is unsuitable.) There are security concerns as some MIME types represent executable content. Servers can prevent MIME sniffing by sending the X-Content-Type-Options header.

According to Mozilla's documentation:

The X-Content-Type-Options response HTTP header is a marker used by the server to indicate that the MIME types advertised in the Content-Type headers should not be changed and be followed. This allows to opt-out of MIME type sniffing, or, in other words, it is a way to say that the webmasters knew what they were doing.

Therefore adding X-Content-Type-Options: nosniff makes it work.


The browser can process and render the data as it comes in whether data is sent chunked or not. Whether a browser renders the response data is going to be a function of the data structure and what kind of buffering it employs. e.g. Before the browser can render an image, it needs to have the document (or enough of the document), the style sheet, etc.

Chunking is mostly useful when the length of a resource is unknown at the time the resource response is generated (a "Content-Length" can't be included in the response headers) and the server doesn't want to close the connection after the resource is transferred.