How to read large files from google drive into gae blobstore

Question

This is the code that I'm currently using to read files from google drive into the blobstore of google app engine.

private BlobKey getBlobKey(File f, DriveObject driveObject)
            throws IOException, MalformedURLException { 

        Drive service = ((GoogleDrive) driveObject).getService();

        byte[] buffer = new byte[(int) f.getFileSize().intValue()];
        GenericUrl url = new GenericUrl(f.getDownloadUrl());
        HttpResponse response = service.getRequestFactory()
                    .buildGetRequest(url).execute();

        InputStream is = response.getContent();

        FileService fileService = FileServiceFactory.getFileService();
        AppEngineFile file = null;
        boolean lock = true;
        try {
            file = fileService.createNewBlobFile("application/zip");
            FileWriteChannel writeChannel = fileService.openWriteChannel(
                        file, lock);

            int len;
            while ((len = is.read(buffer)) >= 0) {
            ByteBuffer bb = ByteBuffer.wrap(buffer, 0, len);
            writeChannel.write(bb);
            }
            writeChannel.closeFinally();

        } catch (IOException e) {
                // TODO Auto-generated catch block
            e.printStackTrace();
        }

            BlobKey bk = fileService.getBlobKey(file);

        return bk;
    }

This works fine for small files but it gives an error if the file is more that ~25MB. Is there a better way to do what I'm trying to do?

This is the error I see

Uncaught exception from servlet
com.google.appengine.api.urlfetch.ResponseTooLargeException: The response from url https://doc-0o-bs-docs.googleusercontent.com/docs/securesc/97o44sjeam7f23b1flie9m3qqidhrago/97q4akk8bmpub24itqhclo1vakjusu84/1368259200000/14350098165097390874/14350098165097390874/0ByS2XCxCEzq-NUVuajAwWGVoR1U?h=16653014193614665626&e=download&gd=true was too large.
    at com.google.appengine.api.urlfetch.URLFetchServiceImpl.convertApplicationException(URLFetchServiceImpl.java:132)
    at com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch(URLFetchServiceImpl.java:43)
    at com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$Connection.fetchResponse(URLFetchServiceStreamHandler.java:417)
    at com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$Connection.getInputStream(URLFetchServiceStreamHandler.java:296)
    at com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$Connection.getResponseCode(URLFetchServiceStreamHandler.java:149)
    at com.google.api.client.http.javanet.NetHttpResponse.<init>(NetHttpResponse.java:37)
    at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:95)
    at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:980)

I should add that

Drive service = ((GoogleDrive) driveObject).getService();

returns

new Drive.Builder(httpTransport, jsonFactory, cred)
            .build()

Answer 1

You can download the file from Drive in chunks. Use the Range header in your download request as documented in the Google Drive documentation.

Range: bytes=500-999