How http2/http1.1 proxy handle the Transfer-Encoding?

Question

HTTP/2 prohibits connection-specific header fields. The following header fields must not appear: "Connection", "Keep-Alive", "Proxy-Connection", "Transfer-Encoding" and "Upgrade". Additionally, "TE" header field must not include any value other than "trailers".

What I want to ask is, since the HTTP/2 prohibits the Transfer-Encoding header, so how HTTP/2 proxy handle the header Tranfer-Encoding: chunked?

Should the proxy always cache the whole chunked request in memory and add the Content-Length header, send to HTTP/2 server/client?

Answer 1

In HTTP/2, content is always "chunked" because it gets sent in DATA frames, i.e. blocks of bytes that carry the block length along with the end-of-stream flag that signals if the frame is the last one.

In a HTTP/2 to HTTP/1.1 proxy, the proxy has several choices.

A very simple case is to remap each HTTP/2 received DATA frame and send it out as a HTTP/1.1 chunk. Therefore, the proxy will have to add the Transfer-Encoding: chunked header to the HTTP/1.1 side of the communication. Similarly, it can remap each content read into a DATA frame (as opposed to wait/buffer for the whole chunk if it is large).

Another case is to buffer some of the DATA frames received, in the hope that one of them has the end-of-stream flag set. If this happens, then the whole content length is known, and the proxy can add the Content-Length header and send the whole content at once.

Also in the previous case, when the buffer overflows the proxy can add the Transfer-Encoding: chunked header and send a chunk that is the size of the buffer (and not the size of the DATA frame like in the first case).

When the proxy receives the last DATA frame, it chunks the remaining bytes and then send the terminal chunk (the zero-length chunk that signals the end of the chunks).

In the other direction, HTTP/1.1 to HTTP/2, when the proxy receives chunked content, it can simply discard the Transfer-Encoding: chunked header, make a DATA frame out of the received chunk, and send that frame. Eventually it will received the terminal chunk (the zero-length chunk that signals the end of the chunks), and translate that into a DATA frame of length zero with the end-of-stream flag set.

Of course the DATA frame sizes may not be exactly equal to the chunk sizes if there is some buffering or other optimizations performed by the proxy.

Thanks to the fact that HTTP/2 receives/sends DATA frames that contain the end-of-stream flag, a proxy can easily translate to/from HTTP/1.1, mapping the end-of-stream DATA frame to the terminal chunk, and the terminal chunk to an end-of-stream DATA frame.