Traverse a type generated by Azure Storage type provider in F#

Question

I am trying to get my head around type providers in F# and what they can be used for. I have the following problem:

I have a series of JSON objects in Azure Blob Storage stored as follows:

container/YYYY/MM/DD/file.json

I can easily navigate to a specific file for a given date using a type provider. For example, I can access the JSON object as a string for the 5th of May as

type Azure = AzureTypeProvider<"ConnectionString">
let containers  = Azure.Containers.``container``.``2017/``.``05/``.``05/``.``file.json``.Read()

How can I take a user input date string, say "2017-05-05" and get the corresponding JSON object in a type safe way? Should I even be using type providers?

Answer 1

You're coming up against a common "issue" with the nature of many TPs, particularly ones that offer a schema against actual data - because it blends the line between data and types, you need to be aware of when you're working in a mode that works well with static types (i.e. you know at compile time the schema of the blob containers you're working with), or working in a way that's inherently dynamic.

You have a few options here.

1. Fall back to the "native" .NET SDK. Every blob / container has associated AsCloudBlob() or AsCloudContainer() methods, so you can use the TP for the bits that you do know e.g container name, maybe top level folders etc., and then fall back to the native SDK for the weakly-typed bits.

2. Since the latest release of the TP, there's now support for programmatic access in a couple of ways: -

   • You can use indexers to get an unsafe handle to a blob e.g. let blob = Azure.Containers.container.["2017/05/05/file.json"]. There's no guarantee that the blob exists, so you need to check that yourself etc.

   • You can use the TryGetBlockBlob() method, which returns a blob option async - behind the scenes, it will do a check if the blob exists or not, and then return either None, or Some blob.

You can see more examples of all of these alternatives here.

3. If you know up-front the full path you're working with (at compile time - perhaps some well-know paths etc.), you can also use the offline support in the TP to create an explicit blob schema at compile time without needing a real storage account.