Export Bigquery Logs

Question

I want to analyze the activity on BigQuery during the past month.

I went to the cloud console and the (very inconvenient) log viewer. I set up exports to Big-query, and now I can run queries on the logs and analyze the activity. There is even very convenient guide here: https://cloud.google.com/bigquery/audit-logs.

However, all this helps to look at data collected from now on. I need to analyze past month.

Is there a way to export existing logs (rather than new) to Bigquery (or to flat file and later load them to BQ)?

Thanks

Answer 1

While you cannot "backstream" the BigQuery's logs of the past, there is something you can still do, depending on what kind of information you're looking for. If you need information about query jobs (jobs stats, config etc), you can call Jobs: list method of BigQuery API to list all jobs in your project. The data is preserved there for 6 months and if you're project owner, you can list the jobs of all users, regardless who actually ran it.

If you don't want to code anything, you can even use API Explorer to call the method and save the output as json file and then load it back into BigQuery's table.

Sample code to list jobs with BigQuery API. It requires some modification but it should be fairly easy to get it done.