Why does this code so negatively affect my server's performance?

Question

I have a Silverstripe site that deals with very big data. I made an API that returns a very large dump, and I call that API at the front-end by ajax get.

When ajax calling the API, it will take 10 mins for data to return (very long json data and customer accepted that).

While they are waiting for the data return, they open the same site in another tab to do other things, but the site is very slow until the previous ajax request is finished.

Is there anything I can do to avoid everything going unresponsive while waiting for big json data?

Here's the code and an explanation of what it does:

I created a method named geteverything that resides on the web server as below, it accessesses another server (data server) to get data via streaming API (sitting in data server). There's a lot of data, and the data server is slow; my customer doesn't mind the request taking long, they mind how slow everything else becomes. Sessions are used to determine particulars of the request.

protected function geteverything($http, $id) {
    if(($System = DataObject::get_by_id('ESM_System', $id))) {
        if(isset($_GET['AAA']) && isset($_GET['BBB']) && isset($_GET['CCC']) && isset($_GET['DDD'])) {
            /**
              --some condition check and data format for AAA BBB CCC and DDD goes here
            **/
            $request = "http://dataserver/streaming?method=xxx";
            set_time_limit(120);
            $jsonstring = file_get_contents($request);
            echo($jsonstring);
        }
    }
}

How can I fix this, or what else would you need to know in order to help?

Answer 1

The reason it's taking so long is your downloading the entirity of the json to your server THEN sending it all to the user. There's no need to wait for you to get the whole file before you start sending it.

Rather than using file_get_contents make the connection with curl and write the output directly to php://output.

For example, this script will copy http://example.com/ exactly as is:

<?php

    // Initialise cURL. You can specify the URL in curl_setopt instead if you prefer
    $ch = curl_init("http://example.com/");

    // Open a file handler to PHP's output stream
    $fp = fopen('php://output', 'w');    

    // Turn off headers, we don't care about them
    curl_setopt($ch, CURLOPT_HEADER, 0);

    // Tell curl to write the response to the stream
    curl_setopt($ch, CURLOPT_FILE, $fp);

    // Make the request
    curl_exec($ch);

    // close resources
    curl_close($ch);
    fclose($fp);