Using node.js and promise to fetch paginated data

Question

Please keep in mind that I am new to node.js and I am used with android development.

My scenario is like this:

1. Run a query against the database that returns either null or a value
2. Call a web service with that database value, that offers info paginated, meaning that on a call I get a parameter to pass for the next call if there is more info to fetch.
3. After all the items are retrieved, store them in a database table
4. If everything is well, for each item received previously, I need to make another web call and store the retrieved info in another table
5. if fetching any of the data set fails, all data must be reverted from the database

So far, I've tried this:

getAllData: function(){
  self.getMainWebData(null)
     .then(function(result){
       //get secondary data for each result row and insert it into database
   }
}


getMainWebData: function(nextPage){
    return new Promise(function(resolve, reject) {

      module.getWebData(nextPage, function(errorReturned, response, values) {

    if (errorReturned) {
          reject(errorReturned);
        }

        nextPage = response.nextPageValue;
        resolve(values);
      })
    }).then(function(result) {

    //here I need to insert the returned values in database     

      //there's a new page, so fetch the next set of data
      if (nextPage) {
          //call again getMainWebData? 
          self.getMainWebData(nextPage)
      }

    })

There are a few things missing, from what I've tested, getAllData.then fires only one for the first set of items and not for others, so clearly handling the returned data in not right.

LATER EDIT: I've edited the scenario. Given some more research my feeling is that I could use a chain or .then() to perform the operations in a sequence.

Answer 1

Yes it is happening as you are resolving the promise on the first call itself. You should put resolve(value) inside an if statement which checks if more data is needed to be fetched. You will also need to restructure the logic as node is asynchronous. And the above code will not work unless you do change the logic.

Solution 1:

You can either append the paginated response to another variable outside the context of the calls you are making. And later use that value after you are done with the response.

getAllData: function(){
  self.getMainWebData(null)
  .then(function(result){
       // make your database transaction if result is not an error
   }
}

function getList(nextpage, result, callback){
  module.getWebData(nextPage, function(errorReturned, response, values) {
    if(errorReturned)
      callback(errorReturned);
    result.push(values);
    nextPage = response.nextPageValue;
    if(nextPage)
      getList(nextPage, result, callback);
    else
      callback(null, result);
  })
}
getMainWebData: function(nextPage){
  return new Promise(function(resolve, reject) {
    var result = [];
    getList(nextpage, result, function(err, results){
      if(err)
        reject(err);
      else{
        // Here all the items are retrieved, you can store them in a database table
        //  for each item received make your web call and store it into another variable or result set 
        // suggestion is to make the database transaction only after you have retrieved all your data 
        // other wise it will include database rollback which will depend on the database which you are using
        // after all this is done resolve the promise with the returning value
        resolve(results); 
      }
    });
  })    
}

I have not tested it but something like this should work. If problem persists let me know in comments.

Solution 2:

You can remove promises and try the same thing with callback as they are easier to follow and will make sense to the programmers who are familiar with structural languages.

Answer 2

Looking at your problem, I have created a code that would loop through promises. and would only procede if there is more data to be fetched, the stored data would still be available in an array. I hope this help. Dont forget to mark if it helps.

let fetchData = (offset = 0, limit= 10) => {
    let addresses = [...Array(100).keys()];
    return Promise.resolve(addresses.slice(offset, offset + limit))
}
// o => offset & l => limit
let o = 0, l = 10;
let results = [];
let process = p => {
  if (!p) return p;
  return p.then(data => {
    // Process with data here;
    console.log(data);
    // increment the pagination
    o += l;
    results = results.concat(data);
    // while there is data equal to limit set then fetch next page 
    // otherwise return the collected result
    return (data.length == l)? process(fetchAddress(o, l)).then(data => data) : results;
  })
}
process(fetchAddress(o, l))
.then(data => {
    // All the fetched data will be here
}).catch(err => {
// Handle Error here.
// All the retrieved data from database will be available in "results" array 
});

if You want to do it more often I have also created a gist for reference. If You dont want to use any global variable, and want to do it in very functional way. You can check this example. However it requires little more complication.