Call hierarchy of async functions inside a loop?

Question

There's a async call I'm making that queries a database on a service, but this service has a limit of how many it can output at once, so I need to check if it hit its limit through the result it sends and repeat the query until it doesn't.

Synchronous mockup :

var query_results = [];

var limit_hit = true; #While this is true means that the query hit the record limit
var start_from = 0; #Pagination parameter

while (limit_hit) {
    Server.Query(params={start_from : start_from}, callback=function(result){
        limit_hit = result.limit_hit;
        start_from = result.results.length;
        query_result.push(result.results);
    }
}

Obviously the above does not work, I've seen some other questions here about the issue, but they don't mention what to do when you need each iteration to wait for the last one to finish and you don't know before hand the number of iterations.

How can I turn the above asynchronous? I'm open to answers using promise/deferred-like logic, but preferably something clean.

I can probably think of a monstruous and horrible way of doing this using waits/timeouts, but there has to be a clean, clever and modern way to solve it.

Another way is to make a "pre-query" to know the number of features before hand so you know the number of loops, I'm not sure if this is the correct way.

Here we use Dojo sometimes, but the examples I found does not explain what to do when you have an unknown amount of loops https://www.sitepen.com/blog/2015/06/10/dojo-faq-how-can-i-sequence-asynchronous-operations/

Answer 1

although many answers already, still I believe async/await is the cleanest way.

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function

and you might need babel

https://babeljs.io/

JS async logic syntax changed from callback to promise then to async/await, they all do the same thing, when callback nests a lot we need something like a chain, then promise come, when promise goes in loop, we need something make the chain more plain more simple, then async/await come. But not all browsers support the new syntax, so babel come to compile new syntax to old syntax, then you can always code in new syntax.

getData().then((data) => {
  //do something with final data
})

async function getData() {
  var query_results = [];

  var limit_hit = true;
  var start_from = 0;
  
  //when you use await, handle error with try/catch
  try {
    while (limit_hit) {
      const result = await loadPage(start_from)
      limit_hit = result.limit_hit;
      start_from = result.results.length;
      query_result.push(result.results);
    }
  } catch (e) {
    //when loadPage rejects
    console.log(e)
    return null
  }
  return query_result
}

async function loadPage(start_from) {
  //when you use promise, handle error with reject
  return new Promise((resolve, reject) => Server.Query({
    start_from
  }, (result, err) => {
    //error reject
    if (err) {
      reject(err)
      return
    }
    resolve(result)
  }))
}

Answer 2

If you want to use a loop then I think there is no (clean) way to do it without Promises.

A different approach would be the following:

var query_results = [];

var start_from = 0;

funciton myCallback(result) {  
  if(!result) {
    //first call
    Server.Query({ start_from: start_from}, myCallback);
  } else {
    //repeated call
    start_from = result.results.length
    query_result.push(result.results);
    
    if(!result.limit_hit) {
      //limit has not been hit yet
      //repeat the query with new start value
      Server.Query({ start_from: start_from}, myCallback);
    } else {
        //call some callback function here
    }
  }
}

myCallback(null);

You could call this recursive, but since the Query is asynchronous you shouldn't have problems with call stack limits etc.

Using promises in an ES6 environment you could make use of async/await. Im not sure if this is possible with dojo.