Google Drive PHP API - How to stream a large file

Question

I know how to download files all at once, as shown in the example on this page: Google Drive API -Files: get.

However, if I have a very large file, loading it all into memory would be inefficient and resource-hogging. Therefore, I would like to know if it's possible to stream files with the Drive API, only loading bits of the file into memory at a time, and processing them (in some way like writing to a file or writing directly to output). I've read the docs and even looked a bit at the source for the PHP Google Drive SDK, and it seems that there is support for streaming, but I can't figure out how to use it. All help is appreciated.

Answer 1

Yes it is possible to specify how much of the file to get there by getting it a chunk at a time. But i'm not sure if you can actually read what is in the file until its fully downloaded.

Google drive SDK download files - Partial download

Partial download

Partial download involves downloading only a specified portion of a file. You 
can specify the portion of the file you want to download by using a byte range 
with the Range header. For example:
Range: bytes=500-999

I did a quick scan of the php client lib and I'm not sure that it supports it. This may be something that needs to be added to the client lib or its something that you will have to code on your own with out using the client lib.

Answer 2

Downloading a large file all at once is not a good approach. So for example if you a have downloadable file of 1GB, you are actually creating a PHP variable of that size which can result into 'allowed memory size exceeded' error too.

The good idea is to download files in chunks. If you are using official Google SDK for PHP, you can do something similar to the following:-

$tmpFileName = tempnam(sys_get_temp_dir(),NULL);
$fp = fopen($tmpFileName,"wb");

$downloadUrl = $file->getDownloadUrl();
$request = new Google_Http_Request($downloadUrl, 'GET', null, null);
$request->setRequestHeaders(
  array('Range' => 'bytes=' . $start . '-' . $end)
); // you can loop through this line of code by calculating $start / $end with respect to the total size of file. (size of file in bytes)
$httpRequest = $service->getClient()->getAuth()->authenticatedRequest($request);
fwrite($fp, $httpRequest->getResponseBody());
// when executing last three lines of code in a do / while loop, its a good idea to set $httpRequest = NULL after writing the chunk to the file.

Answer 3

Good example in library git repo https://github.com/google/google-api-php-client/blob/master/examples/large-file-download.php

<?php
      /*
       * Copyright 2011 Google Inc.
       *
       * Licensed under the Apache License, Version 2.0 (the "License");
       * you may not use this file except in compliance with the License.
       * You may obtain a copy of the License at
       *
       *     http://www.apache.org/licenses/LICENSE-2.0
       *
       * Unless required by applicable law or agreed to in writing, software
       * distributed under the License is distributed on an "AS IS" BASIS,
       * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
       * See the License for the specific language governing permissions and
       * limitations under the License.
       */

  include_once __DIR__ . '/../vendor/autoload.php';
include_once "templates/base.php";

echo pageHeader("File Download - Downloading a large file");

/*************************************************
 * Ensure you've downloaded your oauth credentials
 ************************************************/
if (!$oauth_credentials = getOAuthCredentialsFile()) {
    echo missingOAuth2CredentialsWarning();
    return;
}

/************************************************
 * The redirect URI is to the current page, e.g:
 * http://localhost:8080/large-file-download.php
 ************************************************/
$redirect_uri = 'http://' . $_SERVER['HTTP_HOST'] . $_SERVER['PHP_SELF'];

$client = new Google_Client();
$client->setAuthConfig($oauth_credentials);
$client->setRedirectUri($redirect_uri);
$client->addScope("https://www.googleapis.com/auth/drive");
$service = new Google_Service_Drive($client);

/************************************************
 * If we have a code back from the OAuth 2.0 flow,
 * we need to exchange that with the
 * Google_Client::fetchAccessTokenWithAuthCode()
 * function. We store the resultant access token
 * bundle in the session, and redirect to ourself.
 ************************************************/
if (isset($_GET['code'])) {
    $token = $client->fetchAccessTokenWithAuthCode($_GET['code']);
    $client->setAccessToken($token);

// store in the session also
$_SESSION['upload_token'] = $token;

// redirect back to the example
header('Location: ' . filter_var($redirect_uri, FILTER_SANITIZE_URL));
}

// set the access token as part of the client
   if (!empty($_SESSION['upload_token'])) {
    $client->setAccessToken($_SESSION['upload_token']);
if ($client->isAccessTokenExpired()) {
unset($_SESSION['upload_token']);
}
} else {
      $authUrl = $client->createAuthUrl();
  }

/************************************************
 * If we're signed in then lets try to download our
 * file.
 ************************************************/
if ($client->getAccessToken()) {
// Check for "Big File" and include the file ID and size
$files = $service->files->listFiles([
'q' => "name='Big File'",
'fields' => 'files(id,size)'
]);

if (count($files) == 0) {
    echo "
    <h3 class='warn'>
    Before you can use this sample, you need to
    <a href='/large-file-upload.php'>upload a large file to Drive</a>.
    </h3>";
    return;
}

// If this is a POST, download the file
if ($_SERVER['REQUEST_METHOD'] == 'POST') {
// Determine the file's size and ID
$fileId = $files[0]->id;
    $fileSize = intval($files[0]->size);

// Get the authorized Guzzle HTTP client
$http = $client->authorize();

// Open a file for writing
$fp = fopen('Big File (downloaded)', 'w');

// Download in 1 MB chunks
$chunkSizeBytes = 1 * 1024 * 1024;
    $chunkStart = 0;

// Iterate over each chunk and write it to our file
   while ($chunkStart < $fileSize) {
    $chunkEnd = $chunkStart + $chunkSizeBytes;
    $response = $http->request(
    'GET',
    sprintf('/drive/v3/files/%s', $fileId),
    [
    'query' => ['alt' => 'media'],
    'headers' => [
    'Range' => sprintf('bytes=%s-%s', $chunkStart, $chunkEnd)
    ]
    ]
    );
    $chunkStart = $chunkEnd + 1;
fwrite($fp, $response->getBody()->getContents());
}
// close the file pointer
fclose($fp);

// redirect back to this example
header('Location: ' . filter_var($redirect_uri . '?downloaded', FILTER_SANITIZE_URL));
}
}
?>

<div class="box">
<?php if (isset($authUrl)): ?>
<div class="request">
<a class='login' href='<?= $authUrl ?>'>Connect Me!</a>
                                                     </div>
                                                       <?php elseif(isset($_GET['downloaded'])): ?>
<div class="shortened">
<p>Your call was successful! Check your filesystem for the file:</p>
                                                                  <p><code><?= __DIR__ . DIRECTORY_SEPARATOR ?>Big File (downloaded)</code></p>
                                                                                                                                             </div>
                                                                                                                                               <?php else: ?>
<form method="POST">
<input type="submit" value="Click here to download a large (20MB) test file" />
</form>
  <?php endif ?>
    </div>

      <?= pageFooter(__FILE__) ?>