How to create an Incremental loading webpage

Question

I'm writing a page dealing with a large amount of data. It would last forever until my resultant page loaded (nearly infinite) because the data returned is so large. Therefore, I need to implement an incrementally loading page like one at thie url:

http://docs.python.org/

Everytime a search term is entered, it will continue to load, and load, until it got some result, it will display incrementally, pretty cool :D.

[edit] I am using python CGI (server) + Jquery (client). I have asked a similar question here :

Display the result on the webpage as soon as the data is available at server

The fact that I am trying to request a script on server ONLY ONE and let the client page incrementally displays out coming results has given me headache. If I don't get in wrong, the long poll, or something like that is not applied to this situation right?

I am trying to do flush() thing but perhaps I am missing something here, I can not make it to work :(, the result always comes to client all at once. Plus getting the first few bytes, first few results are very general term. I would really appreciated with one can give me some running code, cuz I am very confusing now. Thanks so much.

[edit] As I am trying to stick with call-on-only manner, I manage to shut of the mod_deflate of apache2 but so far I failed. I googled this problem and there comes some cases like myself, it's no luck :(.

Answer 1

There are various possible ways to do so, but the basic trick is to have the search program return results to the wire before finishing. This is something usually done by explicitly calling a flush() call or equivalent every so many results.

Now, to present them you can either

• Use AJAX: Return a very small page with javascript that will trigger the search, which would modify the DOM with the results as they are being flushed (or make several calls to the search program with a parameter denoting how many results you want and the offset)

• Use simple HTML: Use HTML that the browsers don't wait to complete render (this translates to avoiding tables, mostly, as tables are usually rendered when the whole table has arrived)

Of course, if we knew what are you programming in, you could get more concrete advice.