Why is AssetManger.list() so slow?

Question

I'm trying to populate a ListView with a mixture of files stored on the SDcard AND stored as assets in the APK. Using TraceView, I can see that the performance of AssetManager.list() is poor in comparison to File.listFiles(), even though I'm using a filename filter for the SDcard.

Here is a simple method that returns all the png files from a folder on the SDcard:

// The folder on SDcard may contain files other than png, so filter them out
private File[] getMatchingFiles(File path) {
File[] flFiles = path.listFiles(new FilenameFilter() {
    public boolean accept(File dir, String name) {
    name = name.toLowerCase();
    return name.endsWith(".png");
    }
});  
return flFiles;
}

I invoke that method here and it takes about 12ms to retrieve 16 files:

final String state = Environment.getExternalStorageState();           
if (Environment.MEDIA_MOUNTED.equals(state)||Environment.MEDIA_SHARED.equals(state)) {
    File path = Environment.getExternalStoragePublicDirectory(getResources().getString(R.string.path_dir));
if (path.exists()){
    File[] files = getMatchingFiles(path); 
        ... 

Whereas the am.list method takes 49ms to retrieve just the names of about 6 files!

// Get all filenames from specific Asset Folder and store them in String array
AssetManager am = getAssets();
String path = getResources().getString(R.string.path_dir);
String[] fileNames = am.list(path);  
...

Can anyone explain why the performance would be so poor? Is the performance proportional to the number of assets stored in the APK? I'm aware that assets are compressed, but I'm only fetching the names of the assets, which I thought would be stored in a table somewhere.

Answer 1

Coverdriven's comment "stored in a table somewhere" inspired me to solve my own problem which I've been putting off for a while.

This doesn't answer the OP but does offer a different approach and it handles subfolders which CommonsWare's solution doesn't unless you go recursive (which of course is another possible solution). It's specifically aimed at apps which have a large number of assets in subfolders.

I added an ANT pre-build target to run this command (I'm on Windows)

dir assets /b /s /A-d > res\raw\assetfiles

This creates a recursive (/s), barebones (/b) listing of all files, excluding directory entries (/A-d) in my assets folder.

I then created this class to statically load the contents of assetfiles into a hashmap, the key of which is the filename and the value the full path

public class AssetFiles {

// create a hashmap of all files referenced in res/raw/assetfiles

/*map of all the contents of assets located in the subfolder with the name specified in FILES_ROOT
the key is the filename without path, the value is the full path relative to FILES_ROOT
includes the root, e.g. harmonics_data/subfolder/file.extension - this can be passed
directly to AssetManager.open()*/
public static HashMap<String, String> assetFiles = new HashMap<String, String>();
public static final String FILES_ROOT = "harmonics_data";

static {

    String line;
    String filename;
    String path;

    try {

        BufferedReader reader = new BufferedReader(new InputStreamReader(TidesPlannerApplication.getContext().getResources().openRawResource(R.raw.assetfiles)));

        while ((line = reader.readLine()) != null) {
            // NB backlash (note the escape) is specific to Windows
            filename = line.substring(line.lastIndexOf("\\")+1);
            path = line.substring(line.lastIndexOf(FILES_ROOT)).replaceAll("\\\\","/");;
            assetFiles.put(filename, path);
        }

    } catch (IOException e) {
        e.printStackTrace();
    }

}

public static boolean exists(String filename){
    return assetFiles.containsKey(filename);
}

public static String getFilename(String filename){
    if (exists(filename)){
        return assetFiles.get(filename);
    } else {
        return "";
    }

}

}

To use it, I simply call AssetFiles.getFilename(filename) which returns the full path which I can pass to AssetManager.open(). Much much faster!

NB. I haven't finished this class and it's not hardened yet so you'll need to add appropriate exception catches and actions. It's also quite specific to my app in that all of my assets are in subfolders which are in turn located in a subfolder of the assets folder (see FILES_ROOT) but easy to adapt to your situation.

Note also the need to replace backslashes, since Windows generates the assetfiles listing, with forward slashes. You could eliminate this on OSX and *nix platforms.

Answer 2

Can anyone explain why the performance would be so poor?

Reading the contents of a ZIP archive (the APK where the assets are located) is slower than reading the contents of a directory on the filesystem, apparently. In the abstract, this is not especially surprising, as I suspect that this would be true for all major operating systems.

Read in that list() data once, then save it somewhere else for faster access (e.g., database), particularly in a form that is optimized for future lookups (e.g., where a simple database query could give you what you want, vs. having to load and "recursively search it" again).

Answer 3

If you have a deep tree of directories in the assets you can detect firstly if an item is file or directory and then call .list() on it (really accelerates the walking through the tree). This is my solution I've discovered for this:

try {
    AssetFileDescriptor desc = getAssets().openFd(path);  // Always throws exception: for directories and for files
    desc.close();  // Never executes
} catch (Exception e) {
    exception_message = e.toString();
}

if (exception_message.endsWith(path)) {  // Exception for directory and for file has different message
    // Directory
} else {
    // File
}

Answer 4

You can approach APK package as it's a ZIP file and read all the entries using Java's builtin ZipFile. It will give you all the file names with their full paths. Perhaps it shouldn't be hard to find which directories you have.

So far this is the fastest approach I've tested.

credit goes to @obastemur's commit on jxcore-android-basics sample project