How to set deadline for BigQuery on Google App Engine

Question

I have a Google App Engine program that calls BigQuery for data.

The query usually takes 3 - 4.5 seconds and is fine but sometimes takes over five seconds and throws this error:

DeadlineExceededError: The API call urlfetch.Fetch() took too long to respond and was cancelled.

This article shows the deadlines and the different kinds of deadline errors.

Is there a way to set the deadline for a BigQuery job to be above 5 seconds? Could not find it in the BigQuery API docs.

Answer 1

BigQuery queries are fast, but often take longer than the default App Engine urlfetch timeout. The BigQuery API is async, so you need to break up the steps into API calls that each are shorter than 5 seconds.

For this situation, I would use the App Engine Task Queue:

1. Make a call to the BigQuery API to insert your job. This returns a JobID.

2. Place a task on the App Engine task queue to check out the status of the BigQuery query job at that ID.

3. If the BigQuery Job Status is not "DONE", place a new task on the queue to check it again.

4. If the Status is "DONE," then make a call using urlfetch to retrieve the results.