What Combine operator/approach can be used to load all pages of "paged API"?

Question

I am working with a web API which delivers results up to a given limit (pageSize parameter of the request). If the number of results surpasses this limit, the response message is pre-populated with an URL to which the follow-up request can be made to fetch more results. If there are even more results, this is again indicated in the same manner.

My intend is to fetch all results at once.

---

Currently I have something like the following request and response structures:

// Request structure
struct TvShowsSearchRequest {
    let q: String
    let pageSize: Int?
}

// Response structure
struct TvShowsSearchResponse: Decodable {
    let next: String?
    let total : Int
    let searchTerm : String
    let searchResultListShow: [SearchResult]?
}

When resolving the problem 'old style' using completion handlers, I had to write a handler, which is triggering a 'handle more' request with the URL of the response:

func handleResponse(request: TvShowsSearchRequest, result: Result<TvShowsSearchResponse, Error>) -> Void {
    switch result {
    case .failure(let error):
        fatalError(error.localizedDescription)
    case .success(let value):
        print("> Total number of shows matching the query: \(value.total)")
        print("> Number of shows fetched: \(value.searchResultListShow?.count ?? 0)")

        if let moreUrl = value.next {
            print("> URL to fetch more entries \(moreUrl)")

            // start recursion here: a new request, calling the same completion handler...
            dataProvider.handleMore(request, nextUrl: moreUrl, completion: handleResponse)
        }
    }
}

let request = TvShowsSearchRequest(query: "A", pageSize: 50)
dataProvider.send(request, completion: handleResponse)

Internally the send and handleMore functions are both calling the same internalSend which is taking the request and the url, to call afterwards URLSession.dataTask(...), check for HTTP errors, decode the response and call the completion block.

---

Now I want to use the Combine framework and use a Publisher which is providing the paged responses automatically, without the need to call for another Publisher.

I have therefore written a requestPublisher function which takes request and the (initial) url and returns a URLSession.dataTaskPublisher which checks for HTTP errors (using tryMap), decode the response.

Now I have to ensure that the Publisher automatically "renews" itself whenever the last emitted value had a valid next URL and the completion event occurs.

I've found that there is a Publisher.append method which would exactly do this, but the problem I had so far: I want to append only under a certain condition (=valid next).

The following pseudo-code illustrates it:

func requestPublisher(for request: TvShowsSearchRequest, with url: URL) -> AnyPublisher<TvShowsSearchResponse, Error> {
    // ... build urlRequest, skipped here ...

    let apiCall = self.session.dataTaskPublisher(for: urlRequest)
        .tryMap { data, response -> Data in
            guard let httpResponse = response as? HTTPURLResponse else {
                throw APIError.server(message: "No HTTP response received")
            }
            if !(200...299).contains(httpResponse.statusCode) {
                throw APIError.server(message: "Server respondend with status: \(httpResponse.statusCode)")
            }
            return data
        }
        .decode(type: TvShowsSearchResponse.self, decoder: JSONDecoder())
        .eraseToAnyPublisher()
    return apiCall
}


// Here I'm trying to use the Combine approach

var moreURL : String?

dataProvider.requestPublisher(request)
    .handleEvents(receiveOutput: {
        moreURL = $0.next   // remember the "next" to fetch more data
    })
    .append(dataProvider.requestPublisher(request, next: moreURL))  // this does not work, because moreUrl was not prepared at the time of creation!!
    .sink(receiveCompletion: { print($0) }, receiveValue: { print($0) })
    .store(in: &cancellableSet)

I suppose there are people out there who have already resolved this problem in a reactive way. Whenever I find a doable solution, it involves again recursion. I don't think this is how a proper solution should look like.

I'm looking for a Publisher which is sending the responses, without me providing a callback function. Probably there must be a solution using Publisher of Publishers, but I'm not yet understanding it.

---

After the comment of @kishanvekariya I've tried to build everything with multiple publishers:

1. The mainRequest publisher which is getting the response to the "main" request.

2. A new urlPublisher which is receiving all the next URLs of the "main" or any follow-up requests.

3. A new moreRequest publisher which is fetching for each value of urlPublisher a new request, sending all next URLs back to the urlPublisher.

Then I tried to attach the moreRequest publisher to the mainRequest with append.

var urlPublisher = PassthroughSubject<String, Error>()

var moreRequest = urlPublisher
    .flatMap {
        return dataProvider.requestPublisher(request, next: $0)
            .handleEvents(receiveOutput: {
                if let moreURL = $0.next {
                    urlPublisher.send(moreURL)
                }
            })
    }

var mainRequest = dataProvider.requestPublisher(request)
    .handleEvents(receiveOutput: {
        if let moreURL = $0.next {
            urlPublisher.send(moreURL)
        }
    })
    .append(moreRequest)
    .sink(receiveCompletion: { print($0) }, receiveValue: { print($0) })
    .store(in: &cancellableSet)

But this still does not work... I always get the result of the "main" request. All follow up requests are missing.

Answer 1

It seems that I've found the solution myself.

The idea is, that I have an urlPublisher which is initialized with the first URL which is then executed and may feed a next URL to the urlPublisher and by doing so causing a follow-up request.

let url = endpoint(for: request)     // initial URL
let urlPublisher = CurrentValueSubject<URL, Error>(url)

urlPublisher
    .flatMap {
        return dataProvider.requestPublisher(for: request, with: $0)
            .handleEvents(receiveOutput: {
                if let next = $0.next, let moreURL = URL(string: self.transformNextUrl(nextUrl: next)) {
                    urlPublisher.send(moreURL)
                } else {
                    urlPublisher.send(completion: .finished)
                }
            })
    }
    .sink(receiveCompletion: { print($0) }, receiveValue: { print($0) })
    .store(in: &cancellableSet)

So in the end, I used composition of two publishers and flatMap instead of the non-functional append. Probably this is also the solution one would target from the start...