Bigquery stream: 'Failed to insert XX rows due to timeout'

Question

Recent days, our streaming met with

"Failed to insert XX rows. First error: {"errors":[{"reason":"timeout"}],"index":YY}"  

During the past half month of continuous streaming from unchanged data source and program scripts, no such failure has been found before.

project id: red-road-574

Answer 1

Fellow BigQuery team member here.

It looks like our documentation is a bit incorrect, in that we can have partial commit of the rows. We'll fully reject the request if there are invalid rows (structure mismatch), but individual rows may fail to be buffered.

In this case, only the rows indicated failed to commit. If you have an insert id you can simply retry the failed rows, or retry the full request if desired (though each retried row will count against your table quota).

This increased occurrence of these row-level errors is due to a change around how we handle batches of insertions. Previously, the entire request would have encountered a timeout.

Hope that helps. Sean

Answer 2

We just rolled out a change over the weekend that should significantly lower the number of timeouts you encounter.

Over the last couple of weeks, the TableData.insertAll API has seen a massive increase in load from customers. While we should be able to handle that load without a problem, it has exposed a couple of bottlenecks in our system that we've been working to address. From our monitoring, it looks like the latency should be back to normal as of yesterday evening. If you continue to see timeouts, please let us know.