Do loop again when error

Question

I tried to read everything, but i kind of got stuck on one problem. By using bigrquery I create queries to Google BigQuery to get data - unfortunately sometimes my query doesn't work because of a time-out. Q is a SQL-Query and BQ is supposed to store the data downloaded from BigQuery.

Does anybody know how to re-do a loop every time tryCatch gives me an error?

I got this so far:

BQ_Foo <- NULL
tryCatch(
{ 
repeat{
  BQ_Foo <- query_exec(Q_foo,"bigquery")
  if(is.list(BQ_Foo) == TRUE)break }
}
,error=function(e){cat("ERROR : Query not loaded!", "\n")}
)

EDIT:

I tried my first approach again and this time i received this error message:

Error in curl::curl_fetch_memory(url, handle = handle) :
Operation was aborted by an application callback

Does anybody know how to handle this?

Answer 1

Widely based on r2evans answer, here's how to do the same kind of things with withRestarts, with some helps from This blog post:

set.seed(2)
foo <- NULL
operation <- function(x,tries) {
  message(paste("x is",x,"remaining tries",tries))
  withRestarts(
    tryCatch({
      if (runif(1) < x) stop("fail!") else 1
    },
    error=function(e) { invokeRestart("retry")}),
    retry = function() { 
      message("Retrying")
      stopifnot(tries > 0)
      operation(x,tries-1)
    }
  )
}
> operation(0.9,5)
# x is 0.9 remaining tries 5
# Retrying
# x is 0.9 remaining tries 4
# Retrying
# x is 0.9 remaining tries 3
# Retrying
# x is 0.9 remaining tries 2
# Retrying
# x is 0.9 remaining tries 1
[1] 1

It's a kind of recursive call, so you can do whatever you want before calling the function again.

You may do it in the tryCatch error handler the same way, the interest to use restarts handlers is to call a specific function, if you had two tryCatch for which you want nearly same handler behavior then you can add a parameter and use the same handler for each try catch, i.e.:

testfun <- function(x) {
  withRestarts({
    tryCatch(
      {
        ifelse(runif(1) < 0.5,stop("Error Message"),warning("Warning message"))
      },
      warning=function(e) { invokeRestart("logger", level="warning", message=e ) },
      error=function(e) { invokeRestart("logger", level="error", message=e ) } 
    )
    },
    logger = function(level,message) {
      message(date()," [",level,"]: ",message[['message']])
    }
  )
}

Giving:

> set.seed(2)
> testfun()
Fri Jul 29 14:15:11 2016 [error]: Error Message
> testfun()
Fri Jul 29 14:15:12 2016 [warning]: Warning message
> testfun()
Fri Jul 29 14:15:13 2016 [warning]: Warning message
> testfun()
Fri Jul 29 14:15:13 2016 [error]: Error Message

Main interest here is the factorizing of the logger method and to reduce code duplication.

Answer 2

Naïve Solution

You might start with a mildly naïve attempt of putting the repeat/while outside the tryCatch, something like this:

set.seed(2)
foo <- NULL
while (is.null(foo)) {
  foo <- tryCatch({
    if (runif(1) < 0.9) stop("fail!") else 1
  },
  error = function(e) { message("err"); NULL; }
  )
}
# err
# err
# err
# err
message("success: ", foo)
# success: 1

Unfortunately you introduce the possibility that the loop will never return. To protect against this, you can try a counter ...

Less-Naïve Solution

set.seed(2)
foo <- NULL
max_attempts <- 3
counter <- 0
while (is.null(foo) && counter < max_attempts) {
  counter <- counter + 1
  foo <- tryCatch({
    if (runif(1) < 0.9) stop("fail!") else 1
  },
  error = function(e) { message("err"); NULL; }
  )
}
# err
# err
# err
if (is.null(foo)) message("final failure") else message("success: ", foo)
# final failure

Now this is better for you, but it may still inadvertently introduce a denial-of-service "attack" on the server. (Consider "why" the query failed: if it is because the server is temporarily inundated, then you are making things worse by clobbering it even for a few limited requests.) Though it slows you down a little, in the case of a busy server, putting in pauses will ease the burden on the server and possibly give you a better chance of a successful query before failing.

Better Solution

In network parlance, small TCP packets can cause congestion when repeated retried (see Nagle's Algorithm for a quick reference). Using some form of exponential backoff is common, and to guard against two (or more) clients doing exactly same backoff simultaneously, some clients jitter slightly (for example, httr::RETRY).

set.seed(2)
foo <- NULL
max_attempts <- 3
# borrowed from hadley/httr::RETRY
pause_cap <- pause_base <- 1
counter <- 0
while (is.null(foo) && counter < max_attempts) {
  if (counter > 0L) {
    length <- stats::runif(1, max = min(pause_cap, pause_base * (2 ^ counter)))
    message("sleeping ", round(length, 1))
    Sys.sleep(length)
  }
  counter <- counter + 1
  foo <- tryCatch({
    if (runif(1) < 0.9) stop("fail!") else 1
  },
  error = function(e) { message("err"); NULL; }
  )
}
# err
# sleeping 0.7
# err
# sleeping 0.2
if (is.null(foo)) message("final failure") else message("success: ", foo)
# success: 1

Wrap-Up

Somewhat sloppy code, but I hope you get the point. Putting loops on network queries without some form of self-limit can very easily escalate into an inadvertent DOS.