Resumable upload in Drive Rest API V3

Question

I am trying to create a resumable upload session using drive rest API in Android.

As per the documentation the 3 steps needed to be followed are

1. Start a resumable session
2. Save the resumable session URI
3. Upload the file

Step 1 : I use the following code to start the resumable session.

File body = new File();
body.setName(fileName);
body.setMimeType(mimeType);
body.setCreatedTime(modifiedDate);
body.setModifiedTime(modifiedDate);
body.setParents(Collections.singletonList(parentId));

HttpHeaders header = new HttpHeaders();
header.setContentLength(0L);
header.setContentType("application/json; charset=UTF-8");
header.set("X-Upload-Content-Type","image/jpeg");

HttpResponse response= driveObject
                     .files()
                     .create(body)
                     .setRequestHeaders(header)
                     .set("uploadType","resumable")
                     .buildHttpRequest()
                     .execute();

Step 2: Once the execution is complete, I'm printing the response header of the request to see the Location URI

System.out.println(response.getHeader().toString());

The output is as follows

{
    cache-control=[no-cache, no-store, max-age=0, must-revalidate], 
    content-encoding=[gzip], 
    content-type=[application/json; charset=UTF-8], 
    date=[Thu, 06 Oct 2016 02:20:18 GMT], 
    expires=[Mon, 01 Jan 1990 00:00:00 GMT], 
    alt-svc=[quic=":443"; ma=2592000; v="36,35,34,33,32"], 
    pragma=[no-cache], 
    server=[GSE], 
    transfer-encoding=[chunked], 
    vary=[Origin, X-Origin], 
    x-android-received-millis=[1475720421761], 
    x-android-response-source=[NETWORK 200], 
    x-android-sent-millis=[1475720420804], 
    x-content-type-options=[nosniff], 
    x-frame-options=[SAMEORIGIN], 
    x-xss-protection=[1; mode=block]
}

I don't find the Location URI in the response header to start uploading filedata as specified in the documentation nor I find any Java samples to perform resumable upload.

How do I retrieve Location URI as specified in documentation?

Answer 1

I was trying for the better part of a week now and I finally got the resumable uploads to run. It does not work how I expected it would, but it does work.

Don't Use the Drive REST API for Everything

What I learned is that the Google Drive REST API is, as far as I know, not really capable of making chunked uploads. This may be a bug or it may be by design. I may also be too stupid.

But what got me thinking was that I could not see code examples anywhere. Everybody just talked about Http headers all the time. So this is what we're gonna do below. We'll use just the headers.

So here is how you do resumable, chunked uploads with the Google Drive REST API and Android:

0) Initialization

String accountName = "account_name";
GoogleAccountCredential credential = GoogleAccountCredential.usingOAuth2(context, Arrays.asList(SCOPES)).setBackOff(new ExponentialBackOff()).setSelectedAccountName(accountName);

1) Start a Resumable Session

Follow the rules outlined by Google in this document:

POST /upload/drive/v3/files?uploadType=resumable HTTP/1.1
Host: www.googleapis.com
Authorization: Bearer your_auth_token
Content-Length: 38
Content-Type: application/json; charset=UTF-8
X-Upload-Content-Type: image/jpeg
X-Upload-Content-Length: 2000000

{
  "name": "My File"
}

Set all the header fields just like in Google's example. Send it as a POST request. Use your credential variable to get the authorization token. The mime type for X-Upload-Content-Type is not so important, it works without it too (this SO answer provides a nice function to retrieve it from a path). Set the X-Upload-Content-Length to the total length of your file. Set Content-Type to JSON format, since our body will provide the metadata for Google in the JSON format.

Now create your metadata body. I put in a file name and a parent. Set the Content-Length to the length of your body in bytes. Then write your body to the request.getOutputStream() output stream.

URL url = new URL("https://www.googleapis.com/upload/drive/v3/files?uploadType=resumable");
HttpURLConnection request = (HttpURLConnection) url.openConnection();
request.setRequestMethod("POST");
request.setDoInput(true);
request.setDoOutput(true);
request.setRequestProperty("Authorization", "Bearer " + credential.getToken());
request.setRequestProperty("X-Upload-Content-Type", getMimeType(file.getPath()));
request.setRequestProperty("X-Upload-Content-Length", String.format(Locale.ENGLISH, "%d", file.length()));
request.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
String body = "{\"name\": \"" + file.getName() + "\", \"parents\": [\"" + parentId + "\"]}";
request.setRequestProperty("Content-Length", String.format(Locale.ENGLISH, "%d", body.getBytes().length));
OutputStream outputStream = request.getOutputStream();
outputStream.write(body.getBytes());
outputStream.close();
request.connect();

2) Save the Resumable Session URI

Finally, connect() and wait for a response. If the response code is 200, you have successfully initiated a chunked, resumable upload. Now save the location header URI somewhere (database, text file, whatever). You're gonna need it later.

if (request.getResponseCode() == HttpURLConnection.HTTP_OK) {
    String sessionUri = request.getHeaderField("location");
}

3) Upload the File

PUT {session_uri} HTTP/1.1
Host: www.googleapis.com
Content-Length: 524288
Content-Type: image/jpeg
Content-Range: bytes 0-524287/2000000

bytes 0-524288

Put the following code in a loop, until the entire file is uploaded. After every chunk, you will get a response with code 308 and a range header. From this range header, you can read the next chunk start (see (4)).

Content-Type is going to be the mime type again. Content-Length is the number of bytes you upload in this chunk. Content-Range needs to be of the form bytes startByte-EndByte/BytesTotal. You put this in a PUT request.

Then you create a FileInputStream and set the position to your start byte (which you got from your last response range header) and read another chunk into your buffer. This buffer is then written to the connection output stream. Finally, connect().

URL url = new URL(sessionUri);
HttpURLConnection request = (HttpURLConnection) url.openConnection();
request.setRequestMethod("PUT");
request.setDoOutput(true);
request.setConnectTimeout(10000);
request.setRequestProperty("Content-Type", getMimeType(file.getPath()));
long uploadedBytes = chunkSizeInMb * 1024 * 1024;
if (chunkStart + uploadedBytes > file.length()) {
    uploadedBytes = (int) file.length() - chunkStart;
}
request.setRequestProperty("Content-Length", String.format(Locale.ENGLISH, "%d", uploadedBytes));
request.setRequestProperty("Content-Range", "bytes " + chunkStart + "-" + (chunkStart + uploadedBytes - 1) + "/" + file.length());
byte[] buffer = new byte[(int) uploadedBytes];
FileInputStream fileInputStream = new FileInputStream(file);
fileInputStream.getChannel().position(chunkStart);
if (fileInputStream.read(buffer, 0, (int) uploadedBytes) == -1) { /* break, return, exit*/ }
fileInputStream.close();
OutputStream outputStream = request.getOutputStream();
outputStream.write(buffer);
outputStream.close();
request.connect();

4) Handle Response

After this you will get a response with code 308 (if successful). This response contains a range header (mentioned).

HTTP/1.1 308 Resume Incomplete
Content-Length: 0
Range: bytes=0-524287

You split this up and obtain your new chunk start byte.

 String range = chunkUploadConnection.getHeaderField("range");
    int chunkPosition = Long.parseLong(range.substring(range.lastIndexOf("-") + 1, range.length())) + 1;

5) The Response Code Is Not 308?!

It can happen that you get a 5xx response. Your internet connection could fail, the file could be deleted/renamed during upload, etc. etc. Don't worry. As long as you save your session URI and your chunk start byte, you can resume the upload anytime.

In order to do that, send a header of the following form:

PUT {session_uri} HTTP/1.1
Content-Length: 0
Content-Range: bytes */TotalFileLength


URL url = new URL(sessionUri);
HttpURLConnection request = (HttpURLConnection) url.openConnection();
request.setRequestMethod("PUT");
request.setDoOutput(true);
request.setConnectTimeout(10000);
request.setRequestProperty("Content-Length", "0");
request.setRequestProperty("Content-Range", "bytes */" + file.length());
request.connect();

You will then receive a 308 with a range header, from which you can read the last uploaded byte (just as we did above). Take this number and start to loop again.

I hope I could help some of you. If you have any more questions, just ask in the comments and I will edit the answer.

Answer 2

You don't have to care about all this logic. The documentation indeed explain the flow to complete a resumable upload but it's error prone if done "manually".
Fortunately, Google expose a dedicated class to handle such case, i.e the MediaHttpUploader.

This snippet of code do the job of a resumable upload on drive (same thing can be achieved on GCS):

public class Main {

private static final JacksonFactory JSON_FACTORY = new JacksonFactory();
private static final NetHttpTransport HTTP_TRANSPORT = new NetHttpTransport();
private static final MemoryDataStoreFactory DATA_STORE = new MemoryDataStoreFactory();

public static void main(String... args) throws IOException {


    Credential credential = authorize();

    MediaHttpUploader mediaHttpUploader = new MediaHttpUploader(new FileContent("application/json", Paths.get("/path/to/foo.json").toFile()), HTTP_TRANSPORT, credential);
    mediaHttpUploader.setProgressListener(uploader -> System.out.println("progress: " + uploader.getProgress()));
    GenericUrl genericUrl = new GenericUrl(new URL("https://www.googleapis.com/upload/drive/v3/files?name=toto"));
    GenericJson data = new GenericJson();
    data.put("name", "title");


    JsonHttpContent jsonHttpContent = new JsonHttpContent(JSON_FACTORY, data);
    mediaHttpUploader.setMetadata(jsonHttpContent).upload(genericUrl);

    System.out.println("Finished");
}

private static Credential authorize() throws IOException {
    // load client secrets
    try (BufferedReader br = Files.newBufferedReader(Paths.get(Resources.getResource("client_secret.json").getPath()))) {
        GoogleClientSecrets clientSecrets = GoogleClientSecrets.load(JSON_FACTORY, br);

        // set up authorization code flow
        GoogleAuthorizationCodeFlow flow = new GoogleAuthorizationCodeFlow.Builder(
                HTTP_TRANSPORT, JSON_FACTORY, clientSecrets,
                Collections.singleton(DriveScopes.DRIVE))
                .setAccessType("offline")
                .setDataStoreFactory(DATA_STORE).build();
        // authorize
        return new AuthorizationCodeInstalledApp(flow, new LocalServerReceiver()).authorize("user");

    }
}

}

Note that no where we mention location. All the logic is hidden in the MediaHttpUploader class.
So I don't really answer the question (where to find "Location") but I point the fact that this not really needed when using classes from Google library (and I am pretty sure other third party libraries exist to do the same job).

UPDATE: the mediaHttpUploader is what is used under the hood by the Drive v3 client. So we can think about something like that:

      File fileMetadata = new File();
    fileMetadata.setName(UPLOAD_FILE.getName());

    FileContent mediaContent = new FileContent("image/jpeg", UPLOAD_FILE);

    Drive.Files.Create insert = drive.files().create(fileMetadata, mediaContent);
    MediaHttpUploader uploader = insert.getMediaHttpUploader();
    uploader.setDirectUploadEnabled(false);
    uploader.setProgressListener(new FileUploadProgressListener());
    return insert.execute();

Answer 3

Maybe this https://github.com/PiyushXCoder/google-drive-ResumableUpload/blob/master/ResumableUpload.java help you out. However, it was written for servlets but you may easily modify it for android instead.

Well, after getting the comments let me put some extra descriptions.

However, the "ResumableUpload.java" github repo link is well commented and it is enough to make you clear how to perform this upload on google drive. And, you don't actually need to read this long description.

As described in https://developers.google.com/drive/v3/web/resumable-upload by google about how to perform a resumable upload

• We need to make a POST request to inform the server about this upload and get the Session URI to which we'll send our chunks of data for a File. And yeah, we need Access Token for performing this request(Here, the object of Credential has the access token and we'll use it). This request is performed by this method:

    public String requestUploadUrl(HttpServletRequest request, HttpServletResponse response, Credential credential, com.google.api.services.drive.model.File jsonStructure) throws MalformedURLException, IOException
        {
                URL url = new URL("https://www.googleapis.com/upload/drive/v3/files?uploadType=resumable");
                HttpURLConnection req = (HttpURLConnection) url.openConnection();
                req.setRequestMethod("POST");
                req.setDoInput(true);
                req.setDoOutput(true);
                req.setRequestProperty("Authorization", "Bearer " + credential.getAccessToken());
                req.setRequestProperty("X-Upload-Content-Type", jsonStructure.getMimeType());
                req.setRequestProperty("X-Upload-Content-Length", String.valueOf(jsonStructure.getSize()));
                req.setRequestProperty("Content-Type", "application/json; charset=UTF-8");

                String body = "{ \"name\": \""+jsonStructure.getName()+"\" }";
                req.setRequestProperty("Content-Length", String.format(Locale.ENGLISH, "%d", body.getBytes().length));
                OutputStream outputStream = req.getOutputStream();
                outputStream.write(body.getBytes());
                outputStream.close();
                req.connect();

                String sessionUri = null;

                if (req.getResponseCode() == HttpURLConnection.HTTP_OK) {
                    sessionUri = req.getHeaderField("location");
                }
                return sessionUri; 
            }

• Now, when we've got the Session URI we may proceed to send our data for the requested File, chuck wise. And, let's perform PUT requests for each chunk. Each chuck's size should be in multiples of 256KB . The following method may be used for each chunk.

    public int uploadFilePacket(HttpServletRequest request, HttpServletResponse response, String sessionUri, com.google.api.services.drive.model.File jsonStructure, java.io.File file, long chunkStart, long uploadBytes) throws MalformedURLException, IOException
        {
            URL url1 = new URL(sessionUri);
            HttpURLConnection req1 = (HttpURLConnection) url1.openConnection();

            req1.setRequestMethod("PUT");
            req1.setDoOutput(true);
            req1.setDoInput(true);
            req1.setConnectTimeout(10000);

            req1.setRequestProperty("Content-Type", jsonStructure.getMimeType());
            req1.setRequestProperty("Content-Length", String.valueOf(uploadBytes));
            req1.setRequestProperty("Content-Range", "bytes " + chunkStart + "-" + (chunkStart + uploadBytes -1) + "/" + jsonStructure.getSize());

            OutputStream outstream = req1.getOutputStream();

            byte[] buffer = new byte[(int) uploadBytes];
            FileInputStream fileInputStream = new FileInputStream(file);
            fileInputStream.getChannel().position(chunkStart);
            if (fileInputStream.read(buffer, 0, (int) uploadBytes) == -1);
            fileInputStream.close();

            outstream.write(buffer);
            outstream.close();

            req1.connect();

            return req1.getResponseCode();
        }

The following method uploads a file dividing it into chunks.

    public void uploadFile(HttpServletRequest request, HttpServletResponse response, Credential credential, com.google.api.services.drive.model.File jsonStructure, java.io.File file) throws IOException, UploadFileException
        {
            String sessionUrl = requestUploadUrl(request, response, credential, jsonStructure);

            for(long i = 1, j = CHUNK_LIMIT;i = jsonStructure.getSize())
                {
                    j = jsonStructure.getSize() - i + 1;
                }
                int responseCode = uploadFilePacket(request, response, sessionUrl, jsonStructure, file, i-1, j);
                if(!(responseCode == OK || responseCode == CREATED || responseCode == INCOMPLETE)) throw new UploadFileException(responseCode);
            }
        }

That's all.