Ajax call inside each loop

Question

I found many posts with this topic. But the solutions I found is not much unsuitable for me. Some experts advised to change code structure, but I am not sure how can I do that.

What I want:
1) Get a list of movie from SQL database
2) Fetch information from a website for each movie

Problem I face: PHP MAX_TIMEOUT occurs.

Solution I thought: call async req for each movie, separately

Bottleneck: Too many async requests

Can you please advice how to implement that (if possible only JS, not jquery please)?

Some solutions from web:

1) Use ASYNC = FALSE.... I don't want to use SYNC req, pointless of using Ajax then
2) Collect all data, then make Ajax call once ... well, I did that first .. but it is a long script (fetching movie info from web), so ultimately causing PHP MAX_TIMEOUT
3) increase PHP MAX_TIMEOUT ... not feasible, I don't know how much to increase.

JS

function loadData(mArray){
    mArray = [{"movieid":"1","title":"10 Things I Hate About You"},{"movieid":"2","title":"100 Girls"}]; // TO SIMLYFY, I PUT THIS CODE HERE .. NORMALLY I GET THIS ARRAY USING ANOTHER AJAX CALL
    for (var i = 0; i < mArray.length; i++) { 
        var obj = mArray[i];
        webAjaxcall(obj["mid"],obj["title"]);  // DEFINITELY NOT A GOOD IDEA
    }
    return true;
}

function webAjaxcall(mid,title){
    var xmlhttp=new XMLHttpRequest();
    xmlhttp.onreadystatechange=function(){
        if (xmlhttp.readyState==4 && xmlhttp.status==200){
            //DO SOMETHING
        }
    }
    xmlhttp.open("POST","file2.php",true);
    xmlhttp.setRequestHeader("Content-type","application/x-www-form-urlencoded");
    params = "title="+title+"&mid="+mid;
    xmlhttp.send(params);
}

Just in case anybody wants to know how I populate the JS array:

FILE 1

$sql = "SELECT `movieid`,`title` FROM movielist";
    $result = mysql_query($sql) or die(mysql_error());
    while($row=mysql_fetch_assoc($result)){
    $output[] = $row;
    }
    exit(json_encode($output));

FILE 2

$json=file_get_contents("http://www.website.com/?t=".rawurlencode($moviename));
$info=json_decode($json);
DO SOMETHING

AJAX TO GET MOVIELIST

var xmlhttp=new XMLHttpRequest();
var myarr;
xmlhttp.onreadystatechange=function(){
    if (xmlhttp.readyState==4 && xmlhttp.status==200){
        myarr = xmlhttp.responseText;
        loadData(JSON.parse(myarr));
    }
}
xmlhttp.open("POST","file1.php",true);
xmlhttp.setRequestHeader("Content-type","application/x-www-form-urlencoded");
params = "fname=<?php echo $ses_id;?>";
xmlhttp.send(params);

Answer 1

Note: ASYNC = FALSE means synchronous, which means everything is going to happen in sequence, one call waiting for the previous, and ultimately results in blocking code.

Solution / Opinion: assuming that the site (or API) where you're pulling data can't handle multiple results in a single request, the only way you'll be able to handle this volume of looping ajax requests is to cache the ajax results directly in your SQL db:

::pseudo-architecture::

Let's assume the following PHP files:

index.php

• Displays your results
• Handles loop logic to display your movies using a single SQL query
• Results that are not cached display a "loading" indicator
• Write a $(document).ready() function that loops through all the "not-cached" movies, asynchronously calls get.php with appropriate GET parameters for each entry that wasn't already cached. This way it doesn't affect the page load time, as it occurs after the page has already loaded.

::pseudocode::

for movie in movies
    if object has cached data and date retrieved is less than [some time ago]
        return data from SQL db
    else
        display a "caching in progress" notification for that title
        send GET request to get.php

Note: you might need to queue/delay your requests to get.php depending on how powerful your server is, lest you get 1000 separate threads running at once.

get.php

• Example url: http://www.yoursite.com/get.php?name=superman&year=1980
• Retrieves $_GET parameters sent from your index.php's ajax loop and forwards a request to your 3rd party API/website.
• You want this to be asynchronous, throwing a connection-close ASAP (see how to do that)
• Think of this like an API "proxy"

::pseudocode::

send 200 ok status code and connection-close header
get $_GET parameters
retrieve API data for your movie by sending $_GET parameters
cache to your SQL db once data is returned

Ultimately, the page loads in realtime, and in order to see the new data, you'd need to refresh the page (or if you want to get REALLY fancy, do something with WebSockets that notifies the client).