Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Python Azure blob storage upload file bigger then 64 MB

From the sample code, I can upload 64MB, without any problem:

   myblob = open(r'task1.txt', 'r').read()
   blob_service.put_blob('mycontainer', 'myblob', myblob, x_ms_blob_type='BlockBlob')

What if I want to upload bigger size?

Thank you

like image 226
Alvin Avatar asked Mar 07 '13 09:03

Alvin


People also ask

How to block blob in Azure Storage?

Upload a file to block blob. List blobs. Download a blob to file. Delete a blob. Delete the container. If you don't have an Azure subscription, create a free account before you begin. Step 1 : Create a new general-purpose Storage Account to use for this tutorial. Go to the Azure Portal and log in using your Azure account.

What if my Blob is larger than 64 MB?

If your blob is larger than 64 MB, you must upload it as a set of blocks. For more information, see the Put Block (REST API) and Put Block List (REST API) operations.

What is the maximum upload size for a block blob?

The maximum upload size for a block blob is 64 MB. If your blob is larger than 64 MB, you must upload it as a set of blocks. For more information, see the Put Block (REST API) and Put Block List (REST API) operations. It's not necessary to call Put Blob if you upload the blob as a set of blocks.

How do I use Azure Blob storage client library V12 for Python?

This section walks you through preparing a project to work with the Azure Blob Storage client library v12 for Python. Create a Python application named blob-quickstart-v12. In a console window (such as cmd, PowerShell, or Bash), create a new directory for the project. Switch to the newly created blob-quickstart-v12 directory.


2 Answers

I ran into the same problem a few days ago, and was lucky enough to find this. It breaks up the file into chunks and uploads it for you.

I hope this helps. Cheers!

like image 135
user2407064 Avatar answered Nov 02 '22 01:11

user2407064


I'm not a Python programmer. But a few extra tips I can offer (my stuff is all in C):

Use HTTP PUT operations(comp=block option) for as many Blocks (4MB each) as required for your file, and then use a final PUT Block List (comp=blocklist option) that coalesces the Blocks. If your Block uploads fail or you need to abort, the cleanup for deleting the partial set of Blocks previously uploaded is a DELETE command for the file you are looking to create, but this appears supported by the 2013-08-15 version only (Someone from the Azure support should confirm this).

If you need to add Meta information, an additional PUT operation (with the comp=metadata prefix) is what I do when using the Block List method. There might be a more efficient way to tag the meta information without requiring an additional PUT, but I'm not aware of it.

like image 38
perplexed Avatar answered Nov 02 '22 00:11

perplexed