Azure File System - Can I "Watch" or only poll?

Question

I am an experienced windows C# developer, but new to the world of Azure, and so trying to figure out a "best practice" as I implement one or more Azure Cloud Services.

I have a number of (external, and outside of my control) sources that can all save files to a folder (or possibly a set of folders). In the current state of my system under Windows, I have a FileSystemWatcher set up to monitor a folder and raise an event when a file appears there.

In the world of Azure, what is the equivalent way to do this? Or is there?

I am aware I could create a timer (or sleep) to pass some time (say 30 seconds), and poll the folder, but I'm just not sure that's the "best" way in a cloud environment.

It is important to note that I have no control over the inputs - in other words the files are saved by an external device over which I have no control; so I can't, for example, push a message onto a queue when the file is saved, and respond to that message...

Although, in the end, that's the goal... So I intend to have a "Watcher" service which will (via events or polling) detect the presence of one or more files, and push a message onto the appropriate queue for the next step in my workflow to respond to.

It should be noted that I am using VS2015, and the latest Azure SDK stuff, so I'm not limited by anything legacy.

What I have so far is basically this (a snippet of a larger code base):

storageAccount = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("StorageConnectionString"));

// Create a CloudFileClient object for credentialed access to File storage.
fileClient = storageAccount.CreateCloudFileClient();

// Obtain the file share name from the config file
string sharenameString = CloudConfigurationManager.GetSetting("NLRB.Scanning.FileSharename");

// Get a reference to the file share.
share = fileClient.GetShareReference(sharenameString);

// Ensure that the share exists.
if (share.Exists())
{
    Trace.WriteLine("Share exists.");
    // Get a reference to the root directory for the share.
    rootDir = share.GetRootDirectoryReference();

    //Here is where I want to start watching the folder represented by rootDir...
}

Thanks in advance.

Answer 1

If you're using an attached disk (or local scratch disk), the behavior would be like on any other Windows machine, so you'd just set up a file watcher accordingly with FileSystemWatcher and deal with callbacks as you normally would.

There's Azure File Service, which is SMB as-a-service and would support any actions you'd be able to do on a regular SMB volume on your local network.

There's Azure blob storage. These can not be watched. You'd have to poll for changes to, say, a blob container.

Answer 2

You could create a loop that polls the root directory periodically using CloudFileDirectory.ListFilesAndDirectories method. https://msdn.microsoft.com/en-us/library/dn723299.aspx

You could also write a small recursive method to call this in sub directories.

To detect differences you can build up an in memory hash map of all files and directories. If you want something like a persistent distributed cache then you can use ie. Redis to keep this list of files/directories. Every time you poll if the file or directory is not in your list then you detected a new file/ directory under root.

You could separate the responsibility of detection and business logic ie. a worker role keeps polling the directory and writes the new files to a queue and the consumer end another worker role/ web service that does the processing with that information.