Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Azure Searching Metadata in blobs

I am try to find a way to bring back only items in blob storage with metadata that matches a particular piece of data. All fields will have a key called 'FlightNo'.

What I want really want is a way to find all files (listBlobs) that contain a match to the metadata, so one level up, then iterate through that set of data, and find further matches as each file has 5 items of metadata.

Here is my very unfriendly code to date.

 foreach (IListBlobItem item in container.ListBlobs(null, false))
        {
            if (item.GetType() == typeof(CloudBlockBlob))
            {

                CloudBlockBlob blob = (CloudBlockBlob)item;

                blob.FetchAttributes();

                foreach (var metaDataItem in blob.Metadata)
                {
                    dictionary.Add(metaDataItem.Key, metaDataItem.Value);
                }

                if (dictionary.Where(r=>r.Key == "FlightNo" && r.Value == FlightNo).Any())
                {
                    if (dictionary.Where(r => r.Key == "FlightDate" && r.Value == FlightDate).Any())
                    {
                        if (dictionary.Where(r => r.Key == "FromAirport" && r.Value == FromAirport).Any())
                        {
                            if (dictionary.Where(r => r.Key == "ToAirport" && r.Value == ToAirport).Any())
                            {
                                if (dictionary.Where(r => r.Key == "ToAirport" && r.Value == ToAirport).Any())
                                {
                                    retList.Add(new BlobStorage()
                                    {
                                        Filename = blob.Name,
                                        BlobType = blob.BlobType.ToString(),
                                        LastModified = (DateTimeOffset)blob.Properties.LastModified,
                                        ContentType = blob.Properties.ContentType,
                                        Length = blob.Properties.Length,
                                        uri = RemoveSecondary(blob.StorageUri.ToString()),
                                        FlightNo = dictionary.Where(r => r.Key == "FlightNo").Select(r => r.Value).SingleOrDefault(),
                                        Fixture = dictionary.Where(r => r.Key == "FixtureNo").Select(r => r.Value).SingleOrDefault(),
                                        FlightDate = dictionary.Where(r => r.Key == "FlightDate").Select(r => r.Value).SingleOrDefault(),
                                        FromAirport = dictionary.Where(r => r.Key == "FromAirport").Select(r => r.Value).SingleOrDefault(),
                                        ToAirport = dictionary.Where(r => r.Key == "ToAirport").Select(r => r.Value).SingleOrDefault()
                                    });

                                }
                            }
                        }
                    }
                }

                dictionary.Clear();
            }
        }

Thanks. Scott

like image 847
scottsanpedro Avatar asked Sep 28 '17 15:09

scottsanpedro


People also ask

How do I find the metadata of Azure Blob storage?

To retrieve metadata, call the GetProperties or GetPropertiesAsync method on your blob or container to populate the Metadata collection, then read the values, as shown in the example below. The GetProperties methods retrieve blob properties and metadata in a single call.

What is blob search?

In computer vision, blob detection methods are aimed at detecting regions in a digital image that differ in properties, such as brightness or color, compared to surrounding regions.


2 Answers

The accepted answer is highly inefficient, looping through and loading every single Blob and their associated Metadata to check for values wouldn't perform very well with any reasonable volume of data.

It is possible to search Blob meta data using Azure Search. A search index can be created that includes Blobs custom meta data.

The following comprehensive articles explain it all:

Indexing Documents in Azure Blob Storage with Azure Search
Searching Blob storage with Azure Search

like image 73
Fishcake Avatar answered Oct 01 '22 12:10

Fishcake


Although still in preview, with Blob Index, you can now do a query search on blob metadata (tags).

You won't need to loop thru all of your blobs until you find what you're looking for.

Here's a snippet from the full article:

Blob Index—a managed secondary index, allowing you to store multi-dimensional object attributes to describe your data objects for Azure Blob storage—is now available in preview. Built on top of blob storage, Blob Index offers consistent reliability, availability, and performance for all your workloads. Blob Index provides native object management and filtering capabilities, which allows you to categorize and find data based on attribute tags set on the data.

like image 31
Peterson Salamat Avatar answered Oct 01 '22 13:10

Peterson Salamat