Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Blob Code download much slower than MS Azure Storage Explorer

I'm downloading a blob from blob storage that is 1GB in size.

If I use MS Azure storage explorer it takes under 10 minutes (I have a 20 megabits down line).

However when I use code:

await blobRef.DownloadToFileAsync("D:\\temp\\data.mdf", FileMode.Create);

(I've also tried to use an in memory stream) it takes over an hour to download 250MB (At which point I killed it). I've done this test multiple times and it happens consistently.

I also monitored the network traffic.

  • Via Storage Exlorer the network traffic downward is around 20Megabits
  • Via code the network traffic downward is around 1Megabit

EDIT: I'm still using an old version of Azure Storage Explorer (1.4.1). But I can confirm new versions are also giving the same results.

like image 477
Murdock Avatar asked Aug 30 '19 14:08

Murdock


People also ask

What is the download speed of Azure Blob Storage?

When object is downloaded directly from Azure Blob Storage to outside of Azure - the speed (tested by wget) is around 0.2 to 0.5 MB/s When the same object is downloaded to a VM in Azure - the speed is around 60 MB/s

How do I manage Azure Storage in the cloud?

Easily manage your Azure storage accounts in the cloud, from Windows, macOS, or Linux, using Azure Storage Explorer. Easily manage your Azure storage accounts in the cloud, from Windows, macOS, or Linux, using Azure Storage Explorer. Azure Storage Explorer – cloud storage management | Microsoft Azure

What is the download speed of a VM in azure?

When the same object is downloaded to a VM in Azure - the speed is around 60 MB/s It may seem like a network issue, but....

What are the benefits of Azure Data Factory?

For example, use the Azure Data Factory extension to move data from other cloud storage services, such as AWS S3, to Azure Storage. Seamlessly view, search and interact with your data and resources using an intuitive interface. Improved accessibility with multiple screen reader options, high-contrast themes and hot keys on Windows and macOS.


2 Answers

You should specify which version of MS Azure Storage explorer your're using.

If you're using some newer versions of 1.9.0 / 1.8.1 / 1.8.0 etc.(please find more details in this link), then Azure Storage Explorer is integrated with azcopy which is using simple commands designed for optimal performance. So you can have a good-performance for downloading / uploading etc.

When using code for downloading / uploading blobs, you can take use of this Microsoft Azure Storage Data Movement Library. This library is based on the core data movement framework that powers AzCopy, which also provides you high-performance uploading, downloading.

like image 192
Ivan Yang Avatar answered Oct 23 '22 05:10

Ivan Yang


I eventually tried 2 solutions proposed by @Ivan and @mjwills:

  • DownloadToFileParallelAsync resulted in 10min 12secs
  • Microsoft Azure Storage Data Movement Library resulted in 9min 35secs

Both solutions much faster than the original DownloadToFileAsync. DownloadToFileParallelAsync is only available in later versions of the library and hence was not available in the one I had installed.

like image 39
Murdock Avatar answered Oct 23 '22 04:10

Murdock