Optimizing an asynchronous find algorithm

Question

I have a series of consecutively-named pages (URLs, like: http://example.com/book/1, http://example.com/book/2, etc.) but I have no way of knowing how many pages there are in advance. I need to retrieve (a particular part of) each page, keep the obtained info in order, miss no page, and request a minimum amount of null pages.

Currently, I have a recursive asynchronous function which is a bit like this:

pages = []

getPage = (page = 1) ->
  xhr.get "http://example.com/book/#{1}", (response) ->
    if isValid response
      pages.push response
      getPage page++
    else
      event.trigger "haveallpages"

getPage()

xhr.get and event.trigger is pseudo-code and are currently jQuery methods (but that may change). isValid is also pseudo-code, in reality the test in defined within the function, but it's complex and not relevant to the question.

This works well but is slow as only one request is processed at a time. What I'm looking for is a way to make better use of the asynchronous nature of XHRs and retrieve the complete list in less time. Is there a pattern which could help me here? Or a better algorithm?

Answer 1

Just fire simultaneous requests while keeping count of them. There is no need to guess the upper bound, simply stop when requests start to fail like in your original code.

This will generate at most concurrency-1 wasted requests:

pages        = []
concurrency  = 5
currentPage  = 0
haveAllPages = false

getPage = (p) ->
  xhr.get "http://example.com/book/#{p}", (response) ->
    if isValid response
      pages.push response
      getPage ++currentPage if not haveAllPages
    else
      haveAllPages = true

while concurrency--
    getPage ++currentPage