Long Polling on HttpClient result and streaming into the CSV file

Question

Question 1: How do I implement the same behavior? But instead of Observable.interval it will be invoked by callbacks.

For example: I have 5000ms interval but my server is extremely slow, and it doesn't get back the result after 5000ms. But the next call is invoked after 5000ms. I don't want it like that. I would like after the result is return from the server it will invoke the next call.

Question 2: How do I stream the result immediately to csv file without creating multiple files one after another. For this current implementation I use FileSaver which works in IE11. I would like to continue using it. Is there a way to stream data to file instead of collecting it into the array, cause I have large dataset's. Like 1m rows and so... Example:

const progress = Observable.interval(1000)
  .switchMap(() => this.messageService.getResults(query))
  .map(messageResult => messageResult)
  .subscribe((data: MessagesResult) => {
    inProcess = true;
    if (!data.isMoreResults && data.auditMessageList.length === 0) {
      this.fileSaver.save(`result.csv`, csvData);
      inProcess = false;
      this.logger.info('Download file finished...');
      progress.unsubscribe();
    }
    const start = this.filterModel.offset * this.filterModel.limit;
    const rows = [...csvData];
    rows.splice(start, 0, ...data.auditMessageList);
    csvData = rows;
    if (inProcess) {
      this.logger.info('Exporting in progress...');
    }
    query.offset++;
  }, error => this.logger.error(error));

}

Answer 1

As you have found out using Observable.interval won't 'wait' for the rest of the stream.

I generally use repeatWhen with delay

const progress = Observable.defer(() => this.messageService.getResults(query))
  .repeatWhen(notifications => notifications.delay(1000)) 
  ...

Here is working example: https://jsfiddle.net/a0rz6nLv/19/

I don't understand the rest of you code very well.

Don't use progress.unsubscribe(); in subscribe method. Instead consider using takeWhile or takeUntil - both will complete the observable for you.

.takeWhile(data => data.isMoreResults  data.auditMessageList.length > 0)

Also buffering results can be done for example by using reduce or toArray

.reduce((accumulator, data) => data.auditMessageList.concat(accumulator), [])

Side effects are best handled by do operator

.do({
  next: () => {
    inProgress = true;
    this.logger.info('Exporting in progress...');
  },
  complete: () => {
    inProgress = false;
    this.logger.info('Download file finished...');
  }
})

Regarding second question - I don't know - you should be able to stream csv from the server. If you cannot modify the server maybe someone else will know how to do it on client...