How to query Cloud Blobs on Windows Azure Storage

Question

I am using Microsoft.WindowsAzure.StorageClient to manipulate blobs on Azure storage. I have come to the point where the user needs to list the uploaded files and modify/delete them. Since there are many files in one container, what is the best way to query azure storage services to return only the desired files. Also, I would like to be able to return only specific number of blobs so I can implement paging.

There is a method called ListBlobs in the CloudBlobContainer, but it seems like it's returning all of the blobs in the container. That will not work for me.

I searched a lot on this topic and could not find anything useful. This link shows only the basics.

--------- EDIT

My answer below does not retrieve the blobs lazily, but it retrieves all of the blobs in the container and then filters the result. Currently there is no solution for retrieving blobs lazily.

Answer 1

The method ListBlobs retrieves the blobs in that container lazily. So you can write queries against that method that are not executed until you loop (or materialize objects with ToList or some other method) the list.

Things will get clearer with few examples. For those that don't know how to obtain a reference to a container in your Azure Storage Account, I recommend this tutorial.

Order by last modified date and take page number 2 (10 blobs per page):

blobContainer.ListBlobs().OfType<CloudBlob>()          .OrderByDescending(b=>b.Properties.LastModified).Skip(10).Take(10); 

Get specific type of files. This will work if you have set ContentType at the time of upload (which I strongly recomend you do):

blobContainer.ListBlobs().OfType<CloudBlob>()          .Where(b=>b.Properties.ContentType.StartsWith("image")); 

Get .jpg files and order them by file size, assuming you set file names with their extensions:

blobContainer.ListBlobs().OfType<CloudBlob>()     .Where(b=>b.Name.EndsWith(".jpg")).OrderByDescending(b=>b.Properties.Length); 

At last, the query will not be executed until you tell it to:

var blobs = blobContainer.ListBlobs().OfType<CloudBlob>()                           .Where(b=>b.Properties.ContentType.StartsWith("image"));  foreach(var b in blobs) //This line will call the service,                          //execute the query against it and                          //return the desired files {    // do something with each file. Variable b is of type CloudBlob } 

Answer 2

What I've realized about Windows Azure blob storage is that it is bare-bones. As in extremely bare-bones. You should use it only to store documents and associated metadata and then retrieve individual blobs by ID.

I recently migrated an application from MongoDB to Windows Azure blob storage. Coming from MongoDB, I was expecting a bunch of different efficient ways to retrieve documents. After migrating, I now rely on a traditional RDBMS and ElasticSearch to store blob information in a more searchable way.

It's really too bad that Windows Azure blob storage is so limiting. I hope to see much-enhanced searching capabilities in the future (e.g., search by metadata, property, blob name regex, etc.) Additionally, indexes based on map/reduce would be awesome. Microsoft has the chance to convert a lot of folks over from other document storage systems if they did these things.