How to loop over nextPageToken using GoogleDrive's Python Quickstart

Question

My goal is to have a list of all of the items & folders in everyone's Google Drive. I'm starting with trying to make sure the script works on my own. I have read cover-to-cover the Drive REST API documentation, and eventually found this code, which can also be found here.

from __future__ import print_function
import httplib2
import os
import sys

from apiclient import discovery
from oauth2client import client
from oauth2client import tools
from oauth2client.file import Storage

reload(sys)
sys.setdefaultencoding('utf-8')

try:
    import argparse
    flags = argparse.ArgumentParser(parents=[tools.argparser]).parse_args()
except ImportError:
    flags = None

# If modifying these scopes, delete your previously saved credentials
# at ~/.credentials/drive-python-quickstart.json
SCOPES = 'https://www.googleapis.com/auth/drive.metadata.readonly'
CLIENT_SECRET_FILE = 'client_secret.json'
APPLICATION_NAME = 'Drive API Python Quickstart'


def get_credentials():
    """Gets valid user credentials from storage.

    If nothing has been stored, or if the stored credentials are invalid,
    the OAuth2 flow is completed to obtain the new credentials.

    Returns:
        Credentials, the obtained credential.
    """
    home_dir = os.path.expanduser('~')
    credential_dir = os.path.join(home_dir, '.credentials')
    if not os.path.exists(credential_dir):
        os.makedirs(credential_dir)
    credential_path = os.path.join(credential_dir,
                                   'drive-python-quickstart.json')

    store = Storage(credential_path)
    credentials = store.get()
    if not credentials or credentials.invalid:
        flow = client.flow_from_clientsecrets(CLIENT_SECRET_FILE, SCOPES)
        flow.user_agent = APPLICATION_NAME
        if flags:
            credentials = tools.run_flow(flow, store, flags)
        else: # Needed only for compatibility with Python 2.6
            credentials = tools.run(flow, store)
        print('Storing credentials to ' + credential_path)
    return credentials

def main():
    """Shows basic usage of the Google Drive API.

    Creates a Google Drive API service object and outputs the names and IDs
    for up to 10 files.
    """
    credentials = get_credentials()
    http = credentials.authorize(httplib2.Http())
    service = discovery.build('drive', 'v3', http=http)

    results = service.files().list(
        pageSize=1000,fields="nextPageToken, files(mimeType, name)").execute()
    items = results.get('files', [])
    if not items:
        print('No files found.')
    else:
        print('Files:')
        for item in items:
            print('{0} ({1})'.format(item['name'], item['mimeType']))

if __name__ == '__main__':
    main()

My problem is with the nextPageToken, and how to properly use it. The max PageSize is 1000, so I must loop over the nextPageToken, fetch it from the resulting JSON, put it back into the original loop (line 66?), to get another 1000 results. How do I do this?

Answer 1

Let's look the google drive api documentation for the File:list Method

In the fields of your request you are asking the nextPageToken, the result will contain the token for the nextPage (if the nextPage exists). The result will be something like this :

{
 ...,
 "nextPageToken": "V1*3|0|XXXXXX",
 "files": [
  {
   ...
  },...
  ]
}

you can extract nextPageToken value like :

token = results.get('nextPageToken', None)

The List method can take the string parameter pageToken :

The token for continuing a previous list request on the next page. This should be set to the value of 'nextPageToken' from the previous response.

Just set the parameter pageToken in the next request to get the next page of results :

    results = service.files().list(
        pageSize=1000,
        pageToken=token,
        fields="nextPageToken, files(mimeType, name)").execute()
    items = results.get('files', [])

Now you can easily make a loop to get all result.

Answer 2

I will try to demonstrate the concept for you but you'll do the implementation in Python. The short answer is, nextPageToken. nextPageTokens enable you to retrieve the results from the next page.

When you perform a GET request, a nextPageToken will always be included in the response so if you had 1000 results but you only wanted to display 20 per page, you can fetch the remaining 980 files using nextPageToken.

Run this URL and you'll see something like:

"kind": "drive#fileList",
 "nextPageToken": "V1*3|0|CjkxOHY2aDdROE9JYkJGWUJEaU5Ybm1OVURSemJTcWFMa2lRQlVJSnVxYmI2YkYzMmhnVHozeWkwRnASBxCqqcG4kis",
 "incompleteSearch": false,

The value of the nextPageToken here is what you use to get to the next page. When you get to the next page and you have more results, a new nextPageToken will be generated for you until you view/get all the results (980-1000).