Authenticating Google API with a service account with Java API

Question

I am attempting to use oauth API to authenticate a google service account through the Java API. I am hoping to use it to access Google Bigquery. I get an "invalid grant" returned from my API requests.

Here is the code, which is a copy of a basic authentication example (which wasn't for Bigquery.. but another Google API):

  /** Global instance of the HTTP transport. */
  private static final HttpTransport HTTP_TRANSPORT = new NetHttpTransport();

  /** Global instance of the JSON factory. */
  private static final JsonFactory JSON_FACTORY = new JacksonFactory();

  private static Bigquery bigquery;  

public ServiceAccountExample() {

      try {
          try {

            GoogleCredential credential = new  GoogleCredential.Builder().setTransport(HTTP_TRANSPORT)
                .setJsonFactory(JSON_FACTORY)
                .setServiceAccountId(SERVICE_ACCOUNT_EMAIL)
                .setServiceAccountScopes(BigqueryScopes.BIGQUERY)
                .setServiceAccountPrivateKeyFromP12File(new File("GoogleBigQuery-privatekey.p12"))
                //.setRefreshListeners(refreshListeners)
                //.setServiceAccountUser("email.com")
                .build();


            credential.refreshToken();

            bigquery = new Bigquery.Builder(HTTP_TRANSPORT, JSON_FACTORY, credential)
                        //.setApplicationName("GoogleBigQuery/1.0")
                        .build();

            listDatasets(bigquery, "publicdata");

            return;
          } catch (IOException e) {
            System.err.println(e.getMessage());
          }
        } catch (Throwable t) {
          t.printStackTrace();
        }

}

SERVICE_ACCOUNT_EMAIL is the email address of the form: [email protected]

If I remove the credential.refreshToken() line, it fails in the listsDatasets on the first call to Bigquery... Otherwise it fails on credential.refreshToken().. with the same error.

Does BigQuery not accept Service Account authentication?

I believe I have done everything correctly through the API Console. I have:

• Turned access to the Big Query API on.
• Created a service account under the "API Access" tab.
• Downloaded my private key (which is referenced from the code above).
• Given my service account user "can edit" access under the "Teams" tab.
• Enabled Billing.

Have I missed anything? Is there anything else I need to do?

Thanks..

Answer 1

The service account authorization method works just fine with BigQuery. You don't need to call credential.refreshToken(). Make sure that your private key file can be read from your application. Here is an example:

import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.jackson.JacksonFactory;

import com.google.api.services.bigquery.Bigquery;
import com.google.api.services.bigquery.BigqueryScopes;
import com.google.api.services.bigquery.Bigquery.Projects;
import com.google.api.services.bigquery.model.*;

import java.io.File;
import java.io.IOException;
import java.security.GeneralSecurityException;
import java.util.List;

public class BigQueryJavaServiceAccount {

  public static void main(String[] args) throws IOException, InterruptedException, GeneralSecurityException {

    final HttpTransport TRANSPORT = new NetHttpTransport();
    final JsonFactory JSON_FACTORY = new JacksonFactory();

    GoogleCredential credential = new  GoogleCredential.Builder()
      .setTransport(TRANSPORT)
      .setJsonFactory(JSON_FACTORY)
      .setServiceAccountId("[email protected]")
      .setServiceAccountScopes(BigqueryScopes.BIGQUERY)
      .setServiceAccountPrivateKeyFromP12File(new File("key.p12"))
      .build();

    Bigquery bigquery = Bigquery.builder(TRANSPORT, JSON_FACTORY)
      .setHttpRequestInitializer(credential)
      .build();

    Projects.List projectListRequest = bigquery.projects().list();
    ProjectList projectList = projectListRequest.execute();
    List<ProjectList.Projects> projects = projectList.getProjects();
    System.out.println("Available projects\n----------------\n");
    for (ProjectList.Projects project : projects) {
      System.out.format("%s\n", project.getProjectReference().getProjectId());
    }
  }
}

Answer 2

To authenticate with a service account key with the Google Cloud BigQuery client libraries (Maven package) you can use Application Default Credentials.

First set the GOOGLE_APPLICATION_CREDENTIALS environment variable to the full path to your service account JSON key file. In your code, create the BigQuery service object using the default instance.

import com.google.cloud.bigquery.BigQuery;
import com.google.cloud.bigquery.BigQueryOptions;
...
BigQuery bigquery = BigQueryOptions.getDefaultInstance().getService();

This is the recommended way, as it makes it possible to use the same code in multiple environments, such as on Google Compute Engine, where the service account is built-in rather than using a key file.

Alternatively, you may wish to construct a credentials object manually from a service account key file and use that create the BigQuery service object.

import com.google.auth.oauth2.GoogleCredentials;
import com.google.auth.oauth2.ServiceAccountCredentials;
import com.google.cloud.bigquery.BigQuery;
import com.google.cloud.bigquery.BigQueryOptions;

import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;

...

// TODO: update to your key path. 
File credentialsPath = new File("service_account.json");

GoogleCredentials credentials;
try (FileInputStream serviceAccountStream = new FileInputStream(credentialsPath)) {
  credentials = ServiceAccountCredentials.fromStream(serviceAccountStream);
}

// Instantiate a client.
BigQuery bigquery =
    BigQueryOptions.newBuilder()
        .setCredentials(credentials)
        .build()
        .getService();