Handle Paging with RxJava

Question

I'm using Retrofit + RxJava on an Android app and am asking myself about how to handle the API pagination to chain calls until all data is being retrieved. Is something like this:

Observable<ApiResponse> getResults(@Query("page") int page);

The ApiResponse object has a simple structure:

class ApiResponse {
    int current;
    Integer next;
    List<ResponseObject> results;
}

The API will return a next value until is last page.

There's some good way to achieve this? Tried to combine some flatMaps(), but had no success.

Answer 1

You could model it recursively:

Observable<ApiResponse> getPageAndNext(int page) {
  return getResults(page)
      .concatMap(new Func1<ApiResponse, Observable<ApiResponse>>() {

        @Override
        public Observable<ApiResponse> call(ApiResponse response) {
          // Terminal case.
          if (response.next == null) {
            return Observable.just(response);
          }
          return Observable.just(response)
              .concatWith(getPageAndNext(response.next));
        }

      });
}

Then, to consume it,

getPageAndNext(0)
    .concatMap(new Func1<ApiResponse, Observable<ResponseObject>>() {

        @Override
        public Observable<ResponseObject> call(ApiResponse response) {
          return Observable.from(response.results);
        }

    })
    .subscribe(new Action1<ResponseObject>() { /** Do something with it */ });

That should get you a stream of ResponseObject that will arrive in order, and most likely arrive in page-size chunks.

Answer 2

Iopar gave a great example.

Just a small addition.
If you want to get all pages in one onNext() call.
It can be helpful when you want to zip this result with one more Observable.
You should write:

private List<String> list = new LinkedList() {
    {
        add("a");
        add("b");
        add("c");
    }
};

int count = 1;

public Observable<List<String>> getAllStrings(int c) {
    return Observable.just(list)
            .concatMap(
                    strings -> {
                        if (c == 3) {
                            return Observable.just(list);
                        } else {
                            count += 1;
                            return Observable.zip(
                                    Observable.just(list),
                                    getAllStrings(count),
                                    (strings1, strings2) -> {
                                        strings1.addAll(strings2);
                                        return strings1;
                                    }
                            );
                        }
                    }
            );
}

Usages:

getAllStrings(0)
        .subscribe(strings -> {
            Log.w(TAG, "call: " + strings);
        });

and you will get:

call: [a, b, c, a, b, c, a, b, c, a, b, c]

Answer 3

I've answered my solution in a similar post: https://stackoverflow.com/a/34378263/143733

The trick or amendment to the solution provided by @Iopar is the inclusion of a 'trigger' Observable that can be emitted by a variety of ways.

In the code I posted, it is emitted once a full page of elements have been processed however it could also occur based on a user clicking a button/scrolling.