How to handle request timeout(408) error in boto?

Question

We are using domain.select() method, that boto provides, to query SimpleDB. For smaller queries(queries involving couple of hours of data), this method works fine. But when I start using multiple threads and longer queries(24 hours of data), it starts timing out, giving following error on stdout:

-------------------------
         4 0 8
...
<?xml version="1.0"?>
<Response><Errors><Error><Code>QueryTimeout</Code><Message>A timeout occurred when attempting to query domain 'd110824' with query expression 'select * from `d110824` where `timestamp` &gt;= '2011-08-24T10:45:56' and `timestamp` &lt; '2011-08-25T10:45:56' and `identifier` = '00063F052C49' order by `timestamp` asc </Message><BoxUsage>0.0055590278</BoxUsage></Error></Errors><RequestID>....</RequestID></Response>

I want to implement a retry mechanism (exponential backoff), when this error is encountered. Boto doesn't throw any exception for this error and simply prints it. To implement a retry mechanism, I need some kind of error code or exception to know that the error has occurred.

Any thoughts on how to achieve this in boto?

Answer 1

Boto will retry on a 503, but not on a 408.

There are several things that will make boto retry, including a 503 (service not available), and some types of HTTP errors when trying to connect. It will use exponential backoff, and try up to 5 times by default. You can change the number of retries by setting num_retries in the .boto config file:

[Boto]
num_retries = 3

I don't know why it doesn't retry on a 408. The AWS docs I've seen recommend doing so.