How can I indicate to the Storage Access Framework that I no longer require the loading animation?

Question

I am writing a DocumentsProvider for Dropbox. I am attempting to follow the Google guidelines for creating a custom provider, as well as Ian Lake's post on Medium for the same.

I am attempting to incorporate the feature within the Storage Access Framework whereby one indicates that there is more data to load.

The relevant portions of my queryChildDocuments() method looks like:

@Override
public Cursor queryChildDocuments(final String parentDocumentId,
                                  final String[] projection,
                                  final String sortOrder)  {

    if (selfPermissionsFailed(getContext())) {
        // Permissions have changed, abort!
        return null;
    }

    // Create a cursor with either the requested fields, or the default projection if "projection" is null.
    final MatrixCursor cursor = new MatrixCursor(projection != null ? projection : getDefaultDocumentProjection()){
        // Indicate we will be batch loading
        @Override
        public Bundle getExtras() {
            Bundle bundle = new Bundle();
            bundle.putBoolean(DocumentsContract.EXTRA_LOADING, true);
            bundle.putString(DocumentsContract.EXTRA_INFO, getContext().getResources().getString(R.string.requesting_data));
            return bundle;
            }

        };

        ListFolderResult result = null;
        DbxClientV2 mDbxClient = DropboxClientFactory.getClient();

        result = mDbxClient.files().listFolderBuilder(parentDocumentId).start();

        if (result.getEntries().size() == 0) {
            // Nothing in the dropbox folder
            Log.d(TAG, "addRowsToQueryChildDocumentsCursor called mDbxClient.files().listFolder() but nothing was there!");
            return;
        }

        // Setup notification so cursor will continue to build
        cursor.setNotificationUri(getContext().getContentResolver(),
                                  getChildDocumentsUri(parentDocumentId));

        while (true) {

            // Load the entries and notify listener
            for (Metadata metadata : result.getEntries()) {

                if (metadata instanceof FolderMetadata) {
                    includeFolder(cursor, (FolderMetadata) metadata);

                } else if (metadata instanceof FileMetadata) {
                    includeFile(cursor, (FileMetadata) metadata);
                }

            }

            // Notify for this batch
getContext().getContentResolver().notifyChange(getChildDocumentsUri(parentDocumentId), null);

            // See if we are ready to exit
            if (!result.getHasMore()) {
                break;
            }
            result = mDbxClient.files().listFolderContinue(result.getCursor());
        }

This all works fine. I get the cursor loaded with data as I expect. What I get "for free" (presumably due to the extras bundle) is that the SAF automatically places a visual at the top of the screen for both the text to the user ("Requesting data"), and an animated bar (on my Samsung Galaxy S7 running API 27) moving back and forth to indicate that the cursor is loading:

My question is - once I exit the fetch loop and so am done loading, how do I programmatically get rid of both the EXTRA_INFO text and the EXTRA_LOADING animation at the top of the screen? I have scoured the APIs and am not seeing anything that looks like a "signal" to tell the SAF that the loading is complete.

The android docs don't discuss this feature much, Ian's Medium post just briefly mentions to send the notification so the cursor knows to refresh itself. Neither have anything to say about the animation.

Answer 1

I have an answer to this question based on reviewing the code in com.android.documentsui as well as other areas of the AOSP to see how a custom DocumentsProvider is called and used:

1. When the contents of a directory is shown in the Picker, it is done so by virtue of a DirectoryFragment instance.
2. The DirectoryFragment ultimately manages an instance of a DirectoryLoader.
3. The DirectoryLoader asynchonously calls a DocumentsProvider to populate a Cursor, which is wrapped in a DirectoryResult instance and handed off to a Model instance which is the underlying data store for the RecyclerView in the DirectoryFragment. Importantly, the Loader hangs on to a reference to this Cursor when it is done - this will come into play when we need to notify the Loader to do another load.
4. The Model receives the DirectoryResult, populates its data structures using the enclosed Cursor and also updates the status of "isLoading" by querying the EXTRA_LOADING key in the getExtras() of the Cursor. It then notifies a listener also managed by the DirectoryFragment that the data has been updated.
5. DirectoryFragment via this listener checks whether the Model indicates EXTRA_LOADING is set TRUE, and if so will display the progress bar, if not it will remove it. It then performs the notifyDataSetChanged() on the adapter associated with the RecyclerView.

The key for our solution is that display/removal of the progress bar comes after the model has updated itself with the return from the loader.

Further, when the Model instance is asked to update itself, it completely clears out the prior data and iterates over the current cursor to fill itself again. This means our "second fetch" should be done only after all data has been retrieved, and it needs to include the complete data set, not just the "second fetch".

Finally - DirectoryLoader essentially registers an inner class with the Cursor as a ContentObserver only after the Cursor has been returned from queryChildDocuments().

THEREFORE, our solution becomes:

Within the DocumentsProvider.queryChildDocuments(), determine whether or not the complete result set can be satisfied in a single pass or not.

If it can, then simply load and return the Cursor and we are done.

If it can not, then:

1. Insure that the getExtras() of the Cursor for the initial load will return TRUE for the EXTRA_LOADING key

2. Gather the initial batch of data and load the Cursor with it, and utilize an internal cache to save this data for the next query (more on why below). We will return this Cursor after the next step and since EXTRA_LOADING is true, the progress bar will appear.

3. Now comes the tricky part. The JavaDoc for queryChildDocuments() says:

If your provider is cloud-based, and you have some data cached or pinned locally, you may return the local data immediately, setting DocumentsContract.EXTRA_LOADING on the Cursor to indicate that you are still fetching additional data. Then, when the network data is available, you can send a change notification to trigger a requery and return the complete contents.

4. The question is where and when does this notification come from? At this point we are deep down in our Provider code, populating a Cursor with the initial load request. The Provider doesn't know anything about a Loader - it is just responding to a queryChildDocuments() call. And at this point, the Loader doesn't know anything about the Cursor - it is just executing a query() into the system which eventually calls our Provider. And once we return the Cursor back to the Loader, there is no further call into the Provider that happens without some kind of external event (like the user clicking on a file or a directory). From DirectoryLoader:

 if (mFeatures.isContentPagingEnabled()) {
     Bundle queryArgs = new Bundle();
     mModel.addQuerySortArgs(queryArgs);

     // TODO: At some point we don't want forced flags to override real paging...
     // and that point is when we have real paging.
     DebugFlags.addForcedPagingArgs(queryArgs);

     cursor = client.query(mUri, null, queryArgs, mSignal);
 } else {
     cursor = client.query(
               mUri, null, null, null, mModel.getDocumentSortQuery(), mSignal);
 }

 if (cursor == null) {
     throw new RemoteException("Provider returned null");
 }

 cursor.registerContentObserver(mObserver);

5. client.query() is done on a class that ultimately calls our Provider. Notice in the above code that right after the Cursor is returned, the Loader registers itself with the Cursor as a ContentObserver using 'mObserver'. mObserver is an instance of an inner class in the Loader that when notified of a content change will cause the loader to requery again.

6. Therefore we need to take two steps. First is since the Loader does not destroy the Cursor it receives from the initial query(), during the initial call to queryChildDocuments() the Provider needs to register the Cursor with the ContentResolver using the Cursor.setNotificationUri() method and pass a Uri that represents the current sub-directory (the parentDocumentId that is passed into queryChildDocuments()):

   cursor.setNotificationUri(getContext().getContentResolver(), DocumentsContract.buildChildDocumentsUri(, parentDocumentId));

7. Then to kick start the Loader again to gather the rest of the data, spawn a separate thread to perform a loop that a) fetches data, b) concatenates it to the cached results used to populate the Cursor in the first query (which is why I said to save it in step 2), and c) notify the Cursor that the data has changed.

8. Return the Cursor back from the initial query. Since the EXTRA_LOADING is set true, the progress bar will appear.

9. Since the Loader registered itself to be notified when content changes, when the Thread spawned in the Provider via step 7 completes fetching, it needs to call notifyChange() on the Resolver using the same Uri value as was registered on the Cursor in step (6):

   getContext().getContentResolver().notifyChange(DocumentsContract.buildChildDocumentsUri(, parentDocumentId), null);

10. The Cursor receives the notification from the Resolver, and in turn notifies the Loader causing it to requery. This time when the Loader queries my Provider, the Provider notes it is a requery and populates the cursor with the current set of what is in the cache. It also has to note whether the thread is still running or not when it grabs the current snapshot of the cache - if so, it sets the getExtras() to indicate loading is still happening. If not it sets GetExtras() to indicate loading is NOT happening so that the progress bar is removed.

11. After the data is fetched by the Thread, the data set will load into the Model and the RecyclerView will refresh. When the thread dies after its last batch fetch, the progress bar will be removed.

Some important tips I learned along the way:

1. On the call to queryChildDocuments(), the provider must decide whether it can get all the entries in one fetch or not and adjust the result of Cursor.getExtras() appropriately. The docs suggest something like this:

MatrixCursor result = new MatrixCursor(projection != null ?
  projection : DEFAULT_DOCUMENT_PROJECTION) {
    @Override
    public Bundle getExtras() {
      Bundle bundle = new Bundle();
      bundle.putBoolean(DocumentsContract.EXTRA_LOADING, true);
      return bundle;
    }
  };

This is fine if you know when you create the Cursor whether you are getting everything in one fetch or not.

If instead you need to create the Cursor, populate it, then adjust after the fact a different pattern is needed, something ilke:

private final Bundle b = new Bundle()
MatrixCursor result = new MatrixCursor(projection != null ?
  projection : DEFAULT_DOCUMENT_PROJECTION) {
    @Override
    public Bundle getExtras() {
      return b;
    }
  };

Then later you can do this:

result.getExtras().putBoolean(DocumentsContract.EXTRA_LOADING, true);

2. If you need to modify the Bundle returned from getExtras() like in the above example, you MUST code getExtras() to have it return something that can be updated like in the example the above. If you don't you can't modify the Bundle instance that is returned from getExtras() by default. This is because by default, getExtras() will return an instance of Bundle.EMPTY, which itself is backed by an ArrayMap.EMPTY, which the ArrayMap class defines in a way that makes the ArrayMap immutable and thus you get a runtime exception if you try and change it.

3. I recognize that there is a very slight time window between the time I start the thread that populates the rest of the contents and I return the initial Cursor back to the Loader. It is theoretically possible that the thread could complete prior to the Loader registering itself with the Cursor. If that were to happen, then even though the thread notifies the Resolver of the change, since the Cursor hasn't been registered as a listener it won't get the message, and the Loader won't get kick started again. It would be good to know a way to insure this can't happen, but I haven't looked into that other than something like delay the thread 250ms or something.

4. Another issue is to handle the condition when a user navigates away from the current directory while the fetch progress is still occurring. This can be checked by the Provider keeping track of the parentDocumentId passed into queryChildDocuments() each time - when they are the same it is a requery. When different it is a new query. On a new query we cancel the thread if it is active and clear the cache, then handle the query.

5. Another issue is to handle is that there can be multiple sources of a requery to the same directory. The first is when the Thread triggers it via the Uri notification after it is done fetching entries for the directory. The others are when the Loader is requested to refresh which can happen in a few ways (the user swipes down on the screen for example). The key to check for is if queryChildDocuments() is called for the same directory and the Thread is not yet complete, then we have gotten a request to reload from some sort of refresh - we respect this by performing a synchronized loading to the cursor from from the current state of the cache, but expect we will get called yet again when the thread finishes.

6. In my testing there was never a time where the same Provider got called in parallel - as the user navigates through the directories, only one directory at a time is requested. Therefore we can satisfy our "batch fetch" with a single thread, and when we detect that a new directory is requested (user moves away from a directory that is taking too long to load for example), then we can cancel the thread and start a new instance of it on the new directory as needed.

I'm posting the relevant parts of my code to show how I did it, some notes on it:

1. My app supports multiple Provider types so I created an abstract class "AbstractStorageProvider" which extends DocumentsProvider in order to encapsulate the common calls that a Provider gets from the system (like queryRoots, queryChildDocuments, etc.) These in turn delegate to classes for each service I want to support (my own for local storage, Dropbox, Spotify, Instagram, etc.) to populate the cursor. I also place in here a standard method to check and make sure the user hasn't changed the Android permission settings on me outside the app, which would cause an Exception to be thrown.
2. Synchronizing access to the internal cache is critical as the Thread will be working in the background populating the cache as multiple calls come in requesting more data.
3. I am posting a relatively "bare-bones" edition of this code for clarity. There are multiple handlers needed in production code for network failures, configuration changes, etc.

My queryChildDocuments() method in my abstract Provider class calls a createDocumentMatrixCursor() method that can be implemented differently depending on the Provider subclass:

    @Override
    public Cursor queryChildDocuments(final String parentDocumentId,
                                      final String[] projection,
                                      final String sortOrder)  {

        if (selfPermissionsFailed(getContext())) {
            return null;
        }
        Log.d(TAG, "queryChildDocuments called for: " + parentDocumentId + ", calling createDocumentMatrixCursor");

        // Create a cursor with either the requested fields, or the default projection if "projection" is null.
        final MatrixCursor cursor = createDocumentMatrixCursor(projection != null ? projection : getDefaultDocumentProjection(), parentDocumentId);

        addRowsToQueryChildDocumentsCursor(cursor, parentDocumentId, projection, sortOrder);

        return cursor;
}

And my DropboxProvider implementation of createDocumentMatrixCursor:

@Override
/**
 * Called to populate a sub-directory of the parent directory. This could be called multiple
 * times for the same directory if (a) the user swipes down on the screen to refresh it, or
 * (b) we previously started a BatchFetcher thread to gather data, and the BatchFetcher 
 * notified our Resolver (which then notifies the Cursor, which then kicks the Loader).
 */
protected MatrixCursor createDocumentMatrixCursor(String[] projection, final String parentDocumentId) {
    MatrixCursor cursor = null;
    final Bundle b = new Bundle();
    cursor = new MatrixCursor(projection != null ? projection : getDefaultDocumentProjection()){
        @Override
        public Bundle getExtras() {
            return b;
        }
    };
    Log.d(TAG, "Creating Document MatrixCursor" );
    if ( !(parentDocumentId.equals(oldParentDocumentId)) ) {
        // Query in new sub-directory requested
        Log.d(TAG, "New query detected for sub-directory with Id: " + parentDocumentId + " old Id was: " + oldParentDocumentId );
        oldParentDocumentId = parentDocumentId;
        // Make sure prior thread is cancelled if it was started
        cancelBatchFetcher();
        // Clear the cache
        metadataCache.clear();

    } else {
        Log.d(TAG, "Requery detected for sub-directory with Id: " + parentDocumentId );
    }
    return cursor;
}

The addrowsToQueryChildDocumentsCursor() method is what my abstract Provider class calls when its queryChildDocuments() method is called and is what the subclass implements and is where all the magic happens to batch fetch large directory contents. For example, my Dropbox provider subclass utilizes Dropbox APIs to get the data it needs and looks like this:

protected void addRowsToQueryChildDocumentsCursor(MatrixCursor cursor,
                                                  final String parentDocumentId,
                                                  String[] projection,
                                                  String sortOrder)  {

    Log.d(TAG, "addRowstoQueryChildDocumentsCursor called for: " + parentDocumentId);

    try {

        if ( DropboxClientFactory.needsInit()) {
            Log.d(TAG, "In addRowsToQueryChildDocumentsCursor, initializing DropboxClientFactory");
            DropboxClientFactory.init(accessToken);
        }
        final ListFolderResult dropBoxQueryResult;
        DbxClientV2 mDbxClient = DropboxClientFactory.getClient();

        if ( isReQuery() ) {
            // We are querying again on the same sub-directory.
            //
            // Call method to populate the cursor with the current status of
            // the pre-loaded data structure. This method will also clear the cache if
            // the thread is done.
            boolean fetcherIsLoading = false;
            synchronized(this) {
                populateResultsToCursor(metadataCache, cursor);
                fetcherIsLoading = fetcherIsLoading();
            }
            if (!fetcherIsLoading) {
                Log.d(TAG, "I believe batchFetcher is no longer loading any data, so clearing the cache");
                // We are here because of the notification from the fetcher, so we are done with
                // this cache.
                metadataCache.clear();
                clearCursorLoadingNotification(cursor);
            } else {
                Log.d(TAG, "I believe batchFetcher is still loading data, so leaving the cache alone.");
                // Indicate we are still loading and bump the loader.
                setCursorForLoadingNotification(cursor, parentDocumentId);
            }

        } else {
            // New query
            if (parentDocumentId.equals(accessToken)) {
                // We are at the Dropbox root
                dropBoxQueryResult = mDbxClient.files().listFolderBuilder("").withLimit(batchSize).start();
            } else {
                dropBoxQueryResult = mDbxClient.files().listFolderBuilder(parentDocumentId).withLimit(batchSize).start();
            }
            Log.d(TAG, "New query fetch got " + dropBoxQueryResult.getEntries().size() + " entries.");

            if (dropBoxQueryResult.getEntries().size() == 0) {
                // Nothing in the dropbox folder
                Log.d(TAG, "I called mDbxClient.files().listFolder() but nothing was there!");
                return;
            }

            // See if we are ready to exit
            if (!dropBoxQueryResult.getHasMore()) {
                // Store our results to the query
                populateResultsToCursor(dropBoxQueryResult.getEntries(), cursor);
                Log.d(TAG, "First fetch got all entries so I'm clearing the cache");
                metadataCache.clear();
                clearCursorLoadingNotification(cursor);
                Log.d(TAG, "Directory retrieval is complete for parentDocumentId: " + parentDocumentId);
            } else {
                // Store our results to both the cache and cursor - cursor for the initial return,
                // cache for when we come back after the Thread finishes
                Log.d(TAG, "Fetched a batch and need to load more for parentDocumentId: " + parentDocumentId);
                populateResultsToCacheAndCursor(dropBoxQueryResult.getEntries(), cursor);

                // Set the getExtras()
                setCursorForLoadingNotification(cursor, parentDocumentId);

                // Register this cursor with the Resolver to get notified by Thread so Cursor will then notify loader to re-load
                Log.d(TAG, "registering cursor for notificationUri on: " + getChildDocumentsUri(parentDocumentId).toString() + " and starting BatchFetcher");
                cursor.setNotificationUri(getContext().getContentResolver(),getChildDocumentsUri(parentDocumentId));
                // Start new thread
                batchFetcher = new BatchFetcher(parentDocumentId, dropBoxQueryResult);
                batchFetcher.start();
            }
        }

    } catch (Exception e) {
        Log.d(TAG, "In addRowsToQueryChildDocumentsCursor got exception, message was: " + e.getMessage());
    }

The thread ("BatchFetcher") handles populating the cache, and notifying the Resolver after each fetch:

private class BatchFetcher extends Thread {
    String mParentDocumentId;
    ListFolderResult mListFolderResult;
    boolean keepFetchin = true;

    BatchFetcher(String parentDocumentId, ListFolderResult listFolderResult) {
        mParentDocumentId = parentDocumentId;
        mListFolderResult = listFolderResult;
    }

    @Override
    public void interrupt() {
        keepFetchin = false;
        super.interrupt();
    }

    public void run() {
        Log.d(TAG, "Starting run() method of BatchFetcher");
        DbxClientV2 mDbxClient = DropboxClientFactory.getClient();
        try {
            mListFolderResult = mDbxClient.files().listFolderContinue(mListFolderResult.getCursor());
            // Double check
            if ( mListFolderResult.getEntries().size() == 0) {
                // Still need to notify so that Loader will cause progress bar to be removed
                getContext().getContentResolver().notifyChange(getChildDocumentsUri(mParentDocumentId), null);
                return;
            }
            while (keepFetchin) {

                populateResultsToCache(mListFolderResult.getEntries());

                if (!mListFolderResult.getHasMore()) {
                    keepFetchin = false;
                } else {
                    mListFolderResult = mDbxClient.files().listFolderContinue(mListFolderResult.getCursor());
                    // Double check
                    if ( mListFolderResult.getEntries().size() == 0) {
                        // Still need to notify so that Loader will cause progress bar to be removed
                        getContext().getContentResolver().notifyChange(getChildDocumentsUri(mParentDocumentId), null);
                        return;
                    }
                }
                // Notify Resolver of change in data, it will contact cursor which will restart loader which will load from cache.
                Log.d(TAG, "BatchFetcher calling contentResolver to notify a change using notificationUri of: " + getChildDocumentsUri(mParentDocumentId).toString());
                getContext().getContentResolver().notifyChange(getChildDocumentsUri(mParentDocumentId), null);
            }
            Log.d(TAG, "Ending run() method of BatchFetcher");
            //TODO - need to have this return "bites" of data so text can be updated.

        } catch (DbxException e) {
            Log.d(TAG, "In BatchFetcher for parentDocumentId: " + mParentDocumentId + " got error, message was; " + e.getMessage());
        }

    }

}