Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Display the result on the webpage as soon as the data is available at server

Tags:

python

cgi

I am writing a cgi page in Python. Let's say a client sends request to my cgi page. My cgi page does the calculation and as soon as it has the first output, it sends back that output to the client, but it will CONTINUE to do the calculation and send other responses AFTER the first response is sent.

Is what I have presented here possible? I ask this question because in my limited knowledge, in a cgi page responses are sent back on one-time basic, once a response is sent, cgi-page stops running. This thing is made on server side or client side, and how do I implement it?

My server is running Apache. Thank you very much.

I have tried a client code from "dbr" in this forum (thanks to him I got the idea of how long-polling works).

<html>
<head>
    <title>BargePoller</title>
    <script src="http://ajax.googleapis.com/ajax/libs/jquery/1.2.6/jquery.min.js" type="text/javascript" charset="utf-8"></script>

    <style type="text/css" media="screen">
      body{ background:#000;color:#fff;font-size:.9em; }
      .msg{ background:#aaa;padding:.2em; border-bottom:1px #000 solid}
      .old{ background-color:#246499;}
      .new{ background-color:#3B9957;}
    .error{ background-color:#992E36;}
    </style>

    <script type="text/javascript" charset="utf-8">
    function addmsg(type, msg){
        /* Simple helper to add a div.
        type is the name of a CSS class (old/new/error).
        msg is the contents of the div */
        $("#messages").append(
            "<div class='msg "+ type +"'>"+ msg +"</div>"
        );
    }

    function waitForMsg(){
        /* This requests the url "msgsrv.php"
        When it complete (or errors)*/
        $.ajax({
            type: "GET",
            url: "msgsrv.php",

            async: true, /* If set to non-async, browser shows page as "Loading.."*/
            cache: false,
            timeout:50000, /* Timeout in ms */

            success: function(data){ /* called when request to barge.php completes */
                addmsg("new", data); /* Add response to a .msg div (with the "new" class)*/
                setTimeout(
                    'waitForMsg()', /* Request next message */
                    1000 /* ..after 1 seconds */
                );
            },
            error: function(XMLHttpRequest, textStatus, errorThrown){
                addmsg("error", textStatus + " (" + errorThrown + ")");
                setTimeout(
                    'waitForMsg()', /* Try again after.. */
                    "15000"); /* milliseconds (15seconds) */
            },
        });
    };

    $(document).ready(function(){
        waitForMsg(); /* Start the inital request */
    });
    </script>
</head>
<body>
    <div id="messages">
        <div class="msg old">
            BargePoll message requester!
        </div>
    </div>
</body>
</html>

And here is my server code:

import sys
if __name__ == "__main__":
    sys.stdout.write("Content-Type: text/html\r\n\r\n")
    print "<html><body>"
    for i in range(10):
        print "<div>%s</div>" % i
        sys.stdout.flush()
    print "</body></html>"

I am expecting my client page to display 1 number at a time (0,1,2,...), but the data always comes out all at once (01234...). Please help me figure it out. Thanks you guys so much.

Just a little out-track, I am trying to use jquery comet plugin, but I couldn't find sufficient documentation though. Helps would be much appreciated. Thanks again :D

[edit] Ok guys, finally thanks to your guides I have managed to make it work. You're right when predict that mod_deflate is the source of all this.

To sum up, what I have done here:

  • For client, make a long poll page as the html code above

  • For server, disable the mod_deflate by: editing file /etc/apache2/mods-available/deflate.conf, comment out the line with text/html part and restart the server. To ensure that Python doesn't buffer the output itself, include #!/usr/bin/python -u in the beginning of the page. Remember to use sys.stdout.flush() after each printing that you want to appear at the client. The effect may not be transparent, should include time.sleep(1) to test. :D

Thanks you guys very much for supporting and helping solving this :D

like image 418
wakandan Avatar asked Dec 09 '09 13:12

wakandan


People also ask

What process is performed by the web browser to get data from the server?

This is called an HTTP request. The web browser looks for the requested website's IP address by translating the URL of the web pages via the Domain Name System (DNS) or by searching through its cache. This process locates the web server where the site's files are hosted.

Which method should you use to read data from the server?

Simply put, the GET method is used to retreive data from a server at the specified resource.

How do I display data from API in HTML?

Approach: First make the necessary JavaScript file, HTML file and CSS file. Then store the API URL in a variable (here api_url). Define a async function (here getapi()) and pass api_url in that function. Define a constant response and store the fetched data by await fetch() method.


2 Answers

Sure.

There's traditional server-driven approach, where the script runs just once, but takes a long time to complete, spitting out bits of page as it goes:

import sys, time

sys.stdout.write('Content-Type: text/html;charset=utf-8\r\n\r\n')

print '<html><body>'
for i in range(10):
    print '<div>%i</div>'%i
    sys.stdout.flush()
    time.sleep(1)

When writing an app to WSGI, this is done by having the application return an iterable which outputs each block it wants sent separately one at a time. I'd really recommend writing to WSGI; you can deploy it through CGI now, but in the future when your app needs better performance you can deploy it through a faster server/interface without having to rewrite.

WSGI-over-CGI example:

import time, wsgiref.handlers

class MyApplication(object):
    def __call__(self, environ, start_response):
        start_response('200 OK', [('Content-Type', 'text/html;charset=utf-8')])
        return self.page()

    def page(self):
        yield '<html><body>'
        for i in range(10):
            yield '<div>%i</div>'%i
            time.sleep(1)

application= MyApplication()
if __name__=='__main__':
    wsgiref.handlers.CGIHandler().run(application)

Note that your web server may foil this approach (for CGI or WSGI) by adding buffering of its own. This typically happens if you're using output-transforming filters like mod_deflate to automatically compress webapp output. You'll need to turn compression off for partial-response-generating scripts.

This limits you to rendering the page bit-by-bit as new data comes in. You can make it prettier by having the client-side take care of altering the page as new data comes in, eg.:

def page(self):
    yield (
        '<html><body><div id="counter">-</div>'
        '<script type="text/javascript">'
        '    function update(n) {'
        '        document.getElementById("counter").firstChild.data= n;'
        '    }'
        '</script>'
    )
    for i in range(10):
        yield '<script type="text/javascript">update(%i);</script>'%i
        time.sleep(1)

This relies on client-side scripting so it might be a good idea to include backup non-script-based final output at the end.

All the while doing this, the page will appear to be still loading. If you don't want that, then you'd need to split the script into a first request that just spits out the static content, including client-side script that checks back with the server using either one XMLHttpRequest that it polls for new data through, or, for the really long-running cases, many XMLHttpRequests each of which returns the status and any new data. This approach is much more complicated as it means you have to run your work process as a background daemon process apart from the web server, and pass data between the daemon and the front-end CGI/WSGI request using eg. pipes or a database.

like image 101
bobince Avatar answered Sep 20 '22 12:09

bobince


Yes thats possible and you don't have do much, as you print data out, server will send it, just to be sure keep flushing stdout

like image 25
Anurag Uniyal Avatar answered Sep 19 '22 12:09

Anurag Uniyal