Very Slow upload/download speed of Firebase Storage

Question

The Story

I am using Firebase Storage in my app to upload large files into the Firebase Storage. Files are mostly videos which can be even larger than 2 GB some times.

What I Have Done

This is what I have done.

UploadTask originalUpload = originalDestination.putFile(Uri.fromFile(originalSource));
            mCurrentUploadTask = originalUpload;
            originalUpload.addOnProgressListener(mOnProgressUpdateListener);
originalUpload.addOnSuccessListener(mOriginalSuccessListener);

Just to inform, I am also converting these sync tasks to async as I need to process everything in the background using,

Tasks.await(originalUpload);

The Problem

The problem is weird and unexpected. The upload/download works perfectly, but it is very slow.

I am on a good internet connection of 1 MBps but these files are never transferred at this speed. It is around 100-150 KBps which is almost 15% of the available speed of my network.

I have tested this several times and on several different networks before making this claim. Nowhere, I found Firebase making full use of the available bandwidth, whereas other non-Firebase downloads/uploads works at full speed.

My app needs to download and upload large files, and therefore cannot afford to transfer at such slow speeds. This was not at all expected from Firebase.

Please let me know if I am doing anything wrong in my implementation or if it is an inherent problem with Firebase?

UPDATE

I have also been experiencing an issue with only the uploads. When ever I am downloading something and I just turn off the Wifi, the download gets cancelled with this error.

com.google.firebase.storage.StorageException: An unknown error occurred, please check the HTTP result code and inner exception for server response.

Read error: ssl=0xb7e7a510: I/O error during system call, Connection timed out
                                                                           javax.net.ssl.SSLException: Read error: ssl=0xb7e7a510: I/O error during system call, Connection timed out
                                                                               at com.android.org.conscrypt.NativeCrypto.SSL_read(Native Method)
                                                                               at com.android.org.conscrypt.OpenSSLSocketImpl$SSLInputStream.read(OpenSSLSocketImpl.java:699)
                                                                               at com.android.okio.Okio$2.read(Okio.java:113)
                                                                               at com.android.okio.RealBufferedSource.read(RealBufferedSource.java:48)
                                                                               at com.android.okhttp.internal.http.HttpConnection$FixedLengthSource.read(HttpConnection.java:446)
                                                                               at com.android.okio.RealBufferedSource$1.read(RealBufferedSource.java:168)
                                                                               at java.io.InputStream.read(InputStream.java:162)
                                                                               at com.google.firebase.storage.FileDownloadTask.run(Unknown Source)
                                                                               at com.google.firebase.storage.StorageTask$5.run(Unknown Source)
                                                                               at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
                                                                               at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
                                                                               at java.lang.Thread.run(Thread.java:818)

But when I do the exact same thing for uploads, it just waits for connection and if I turn on the Wifi again, it reconnects and resumes. Why does the problem only happen with downloading files? Is it a problem with getFile() API again?

FYI, I have not changed the timeout settings for both uploads and downloads. They are at their defaults.

Answer 1

I offer this not to refute your observations, but just as a point of comparison. As a test, I used the code below to upload a ~50MB file and time the progress. The test was run on a Samsung Galaxy S3 (4.4.2) with a home WiFi connection (in California). The average throughput was roughly 760KB/sec.

I ran the same test on other phone devices using the same network with throughput ranging from 300KB/sec to 850KB/sec (Moto X Pure; Marshmallow).

private void uploadTest() {
    // This file is ~50MB
    final File file = new File("/storage/emulated/0/DCIM/Camera/20160821_101145.mp4");
    // Upload the file
    Log.i(TAG, String.format("uploadTest: Starting upload of %5.2fMB file",
            file.length()/1024.0/1024.0));

    FirebaseStorage.getInstance().getReference("test").putFile(Uri.fromFile(file))
            .addOnSuccessListener(new OnSuccessListener<UploadTask.TaskSnapshot>() {
                @Override
                public void onSuccess(UploadTask.TaskSnapshot taskSnapshot) {
                    Log.i(TAG, "onSuccess: Done");
                }
            }).addOnProgressListener(new OnProgressListener<UploadTask.TaskSnapshot>() {
        @Override
        public void onProgress(UploadTask.TaskSnapshot taskSnapshot) {
            Log.i(TAG, String.format("onProgress: %5.2f MB transferred",
                    taskSnapshot.getBytesTransferred()/1024.0/1024.0));
        }
    });
}

This is the logcat produced, with some progress lines deleted to make the list more compact:

08-21 11:39:17.539 18427-18427/com.qbix.test I/MainActivity: uploadTest: Starting upload of 51.79MB file
08-21 11:39:17.559 18427-18427/com.qbix.test I/MainActivity: onProgress:  0.00 MB transferred
08-21 11:39:25.117 18427-18427/com.qbix.test I/MainActivity: onProgress:  5.00 MB transferred
08-21 11:39:31.654 18427-18427/com.qbix.test I/MainActivity: onProgress: 10.00 MB transferred
08-21 11:39:38.711 18427-18427/com.qbix.test I/MainActivity: onProgress: 15.00 MB transferred
08-21 11:39:45.088 18427-18427/com.qbix.test I/MainActivity: onProgress: 20.00 MB transferred
08-21 11:39:51.375 18427-18427/com.qbix.test I/MainActivity: onProgress: 25.00 MB transferred
08-21 11:39:57.411 18427-18427/com.qbix.test I/MainActivity: onProgress: 30.00 MB transferred
08-21 11:40:03.408 18427-18427/com.qbix.test I/MainActivity: onProgress: 35.00 MB transferred
08-21 11:40:10.886 18427-18427/com.qbix.test I/MainActivity: onProgress: 40.00 MB transferred
08-21 11:40:17.233 18427-18427/com.qbix.test I/MainActivity: onProgress: 45.00 MB transferred
08-21 11:40:23.069 18427-18427/com.qbix.test I/MainActivity: onProgress: 50.00 MB transferred
08-21 11:40:25.792 18427-18427/com.qbix.test I/MainActivity: onProgress: 51.79 MB transferred
08-21 11:40:25.792 18427-18427/com.qbix.test I/MainActivity: onSuccess: Done 

Answer 2

I've been doing some testing on my end and I'm trying to get to the bottom of why you see bad perf. I have been able to replicate bad performance when using getStream doing reads of single bytes at a time but this can be fixed by doing a read into a reasonably size byte buffer (256kb buffer). After fixing this, the performance I see is equal to downloading directly with httpsUrlConnection.

Three possibilities pop into my mind:

1) you are seeing timeouts during dns resolution (to our multiple ips) and connection pooling isnt working during uploads. But this would only explain why uploads (which are chunked) would be slow. Downloads are not chunked.

2) you are in a region we have not yet optimized for. We are rolling out local support soon across asia and europe to optimized performance.

3) Maybe you are comparing it with something other than httpsurlconnection? Can you give more details of your test?

Here is my test:

        final long startTime = System.currentTimeMillis();
        // Test via Firebase
        mStorage.child(downloadPath).getStream(new StreamDownloadTask.StreamProcessor() {
               @Override
               public void doInBackground(StreamDownloadTask.TaskSnapshot taskSnapshot,
                      final InputStream inputStream) throws IOException {
                 readStream("FB", inputStream, startTime);
               }
        });

        //Test with AsyncTask
        AsyncTask<Void, Void, Void> task = new AsyncTask<Void, Void, Void>() {
            @Override
            protected Void doInBackground(Void... voids) {
                URL url = null;
                try {
                    url = new URL(uri.toString());
                } catch (MalformedURLException e) {
                    e.printStackTrace();
                }
                try {
                    HttpsURLConnection connection = (HttpsURLConnection) url.openConnection();
                    connection.connect();
                    if (connection.getResponseCode() != HttpURLConnection.HTTP_OK) {
                        return null;
                    }
                    InputStream inputStream = connection.getInputStream();
                    readStream("ASYNC_TASK", inputStream, startTime);
                    inputStream.close();

                }
                catch(IOException e) {

                }

                return null;
            }
        };

        task.execute();

    private void readStream(String desc, InputStream inputStream, long startTime) {
      long bytesRead = 0;
      byte[] buffer = new byte[256*1024];
      int b = 1;
      try {
        while (b > 0) {
            b = inputStream.read(buffer);
            bytesRead += b;
                Log.i("giug", "PROGRESS_" + desc + ": bytesperMS:" +
                        (bytesRead / (System.currentTimeMillis() - startTime)));
        }
        inputStream.close();
      } catch (IOException e) {
          e.printStackTrace();
      }
   }