Facebook Bigpipe Technique Algorithm

Question

I keep studying this flow of the Facebook's bigpipe technique but I have this question.

How this thing is implemented? does the pagelet is received through an ajax request? I keep on searching for the source code of this bigpipe but it points me to a 404 page of github.

Can someone explain this bigpipe in a low level(programming algorithm) way. I'm really interested with this technique.

Thanks in advance

Answer 1

Well, no, the main content and pagelets are received with the same connection. The pagelets are simply streamed as they are generated to the browser, and placed in the document with Javascript.

You can find an open (and simple) BigPipe implementation in PHP here.

Answer 2

I developed a simple page framework recently. The core idea is to separate a page into several features, each of which will be handled in parallel. The output of each feature is an HTML segment, which the framework then assembles by layout configuration. The first version is not perfect. If you get interested, check here https://github.com/chennanfei/Moonlight

Answer 3

Before explaining in detail how Bigpipe works, I will mention I developed a Django extension that implements Bigpipe. Bigpipe-Response.
I will use Bigpipe-Response code to illustrate how Bigpipe works.

Bigpipe is using the initial connection made by the browser, in the following steps:

1. The browser will open an HTTP connection and will request a WEB page.

2. The server will send back the HTML.
   a. Without closing </BODY></HTML> tags. This way to browser assumes that the connection is still open.
   b. The HTML will contain empty <DIV id="pagelet-1"></DIV> elements where bigpipe fill a pagelet content.
   c. Small javascript at the top of the page is included (see step 3)

3. At the top of the page, there is a small javascript function that accepts JSON and populates the page with that JSON content. bigpipe.js

4. While the connection is still open the server will open requests internally to all pagelets. here
5. When a response from a pagelet is available the server will wrap the response data in a JSON (this includes JS, CSS, i18n, links, etc...). and will send it as a parameter for a javascript function call. see here
6. the function renderPagelet inside the Bigpipe.js file will use the JSON data to populate the HTML page.
7. When all pagelets are served, the server will send closing </BODY></HTML> tags.
8. The HTTP connection will be closed here

Hope this helps. for more info, you can refer to the documentation Here.