streaming plain text over HTTP

Question

I wish to stream a set of log messages over HTTP. I want to send the messages one line at a time, possibly with delays between lines, and I want each line to show up in the browser as soon as possible after the server sends it.

My current approach is to set Content-Type to text/plain; charset=UTF-8 in the response and the to just start streaming lines with delays between them as necessary from the server. I am making sure to flush all the relevant output streams after each write.

The behavior I'm observing in Chrome is that it waits until the response is completely finished before showing anything. But the behavior I want is to see each line as it is sent. Is this possible?

I have turned up numerous stackoverflow questions on this topic but none have quite answered my question. I don't think Transfer-Encoding is relevant to me because that seems to be for downloading large files (correct me if I'm wrong).

This is not a question about downloading files because I want the lines to be rendered directly in the browser.

Answer 1

I don't think you can accomplish the "rightest" solution here because of the issues mentioned in the question and answer linked by Ivan. At least my Chrome and Firefox can render the most recent content they receive line by line without any efforts, but, as it was said above, it needs either hacks or changing the requirements to make it more transparent.

The first thing to be done here is fetching but suppressing the first leading n bytes in order to trigger the browser rendering.

If you go with text/plain, you can only rely on how the output text is rendered by a specific browser. To suppress the first dummy block output, you can just render whitespaces since they are not meant to be parsed either by a human or a browser (at least I think so, because you want in-browser output, thus possibly not making it machine-parsable). A trick here is writing Unicode \u200B (zero width space) hoping that a target browser will consume it rendering nothing in the output window. Unfortunately, my Firefox instance does not recognize the character and does render the default unknown character placeholder. However, Chrome completely ignores these characters and visually they look like nothing! And it seems like what you need. So, the general algorithm here is:

• Detect the user agent in order to determine the header block length (you need to know these predefined values).
• Write the UTF-8 BOM (0xEF, 0xBB, 0xBF) to make sure that Chrome won't start the download the remote output to a file.
• Write \u200B character n times, where n is determined in the pre-previous item and flush the output.
• Generate some dummy content with pauses to get new content lines each n seconds flushing immediately after each line.

However, if you would like to have no output rendering issues like that Firefox one for the \u200B character, you might want to switch to text/html. HTML supports markup comments so we can exclude some content from being rendered. This allows to rely on HTML in full, and not a particular browser specifics. Knowing that, the algorithm becomes somewhat different:

• Detect the user agent in order to determine the header block length.
• Render the start of the block with <!--, then some n whitespaces (but at least one as far as I remember; or whatever HTML comment), and then -->. The n should be the length of the block above minus the lengths of the comment begin/end markers.
• Generate some dummy output where each line is HTML-escaped, terminated with <br/> or <br>, and then flushed immediately.

This approach works fine both in Chrome and Firefox to me. If you're fine with some Java, here is some code that implements the said above:

@RestController
@RequestMapping("/messages")
public final class MessagesController {

    private static final List<String> lines = asList(
            "Lorem ipsum dolor sit amet,",
            "consectetur adipiscing elit,",
            "sed do eiusmod tempor incididunt ut labore et dolore magna aliqua."
    );

    @RequestMapping(value = "html", method = GET, produces = "text/html")
    public void getHtml(final HttpServletRequest request, final ServletResponse response)
            throws IOException, InterruptedException {
        render(Renderers.HTML, request, response);
    }

    @RequestMapping(value = "text", method = GET, produces = "text/plain")
    public void getText(final HttpServletRequest request, final ServletResponse response)
            throws IOException, InterruptedException {
        render(Renderers.PLAIN, request, response);
    }

    private static void render(final IRenderer renderer, final HttpServletRequest request, final ServletResponse response)
            throws IOException, InterruptedException {
        final int stubLength = getStubLength(request);
        final ServletOutputStream outputStream = response.getOutputStream();
        renderer.renderStub(stubLength, outputStream);
        renderInfiniteContent(renderer, outputStream);
    }

    private static int getStubLength(final HttpServletRequest request) {
        final String userAgent = request.getHeader("User-Agent");
        if ( userAgent == null ) {
            return 0;
        }
        if ( userAgent.contains("Chrome") ) {
            return 1024;
        }
        if ( userAgent.contains("Firefox") ) {
            return 1024;
        }
        return 0;
    }

    private static void renderInfiniteContent(final IRenderer renderer, final ServletOutputStream outputStream)
            throws IOException, InterruptedException {
        for ( ; ; ) {
            for ( final String line : lines ) {
                renderer.renderLine(line, outputStream);
                sleep(5000);
            }
        }
    }

    private interface IRenderer {

        void renderStub(int length, ServletOutputStream outputStream)
                throws IOException;

        void renderLine(String line, ServletOutputStream outputStream)
                throws IOException;

    }

    private enum Renderers
            implements IRenderer {

        HTML {
            private static final String HTML_PREFIX = "<!-- ";
            private static final String HTML_SUFFIX = " -->";
            private final int HTML_PREFIX_SUFFIX_LENGTH = HTML_PREFIX.length() + HTML_SUFFIX.length();

            @Override
            public void renderStub(final int length, final ServletOutputStream outputStream)
                    throws IOException {
                outputStream.print(HTML_PREFIX);
                for ( int i = 0; i < length - HTML_PREFIX_SUFFIX_LENGTH; i++ ) {
                    outputStream.write('\u0020');
                }
                outputStream.print(HTML_SUFFIX);
                outputStream.flush();
            }

            @Override
            public void renderLine(final String line, final ServletOutputStream outputStream)
                    throws IOException {
                outputStream.print(htmlEscape(line, "UTF-8"));
                outputStream.print("<br/>");
            }
        },

        PLAIN {
            private static final char ZERO_WIDTH_CHAR = '\u200B';
            private final byte[] bom = { (byte) 0xEF, (byte) 0xBB, (byte) 0xBF };

            @Override
            public void renderStub(final int length, final ServletOutputStream outputStream)
                    throws IOException {
                outputStream.write(bom);
                for ( int i = 0; i < length; i++ ) {
                    outputStream.write(ZERO_WIDTH_CHAR);
                }
                outputStream.flush();
            }

            @Override
            public void renderLine(final String line, final ServletOutputStream outputStream)
                    throws IOException {
                outputStream.println(line);
                outputStream.flush();
            }
        }

    }

}

Also, the approach you want to accomplish will not scroll down the browser window. You might want to use a user script in Chrome to scroll down particular URL pages automatically, but, as far as I know, it would not work for text/plain output though.