What is the clean up mechanism for the blobs that WebJobs SDK creates in the AzureWebJobsDashboard connection?

Question

Azure WebJob SDK uses the storage connection string defined in the AzureWebJobsStorage and AzureWebJobsDashboard app settings for its logging and dashboard.

WebJob SDK creates the following blob container in AzureWebJobsStorage:

• azure-webjobs-hosts

WebJob SDK creates the following blob containers in AzureWebJobsDashboard

• azure-jobs-host-output
• azure-webjobs-hosts

Many blobs are created in the above blob containers as the WebJob runs. The containers can be bloated or saturated if there is no clean-up mechanism.

What is the cleanup mechanism for the above blob containers?

Update

The answer below is a workaround. At this point, there is no built-in mechanism to clean up the WebJobs logs. The logs can pile up quite large as the Job runs in a long term. Developers must create the cleanup mechanism on their own. Azure Functions is a good way of implementing such cleanup process. An example is provided in the below answer.

Answer 1

What is the clean up mechanism for the blobs that WebJobs SDK creates in the AzureWebJobsDashboard connection?

I haven’t found a way to do it. There is an open issue on GitHub which related to this topic, but haven’t been closed.

No way to set webjob logging retention policy

In a similar issue on GitHub we found that Azure WebJob SDK have changed a way of saving logs to multi tables of Azure Table Storage. We can easily delete the table per month. For logs writen in Azure Blob Storage haven’t been grouped by month until now.

WebJobs.Logging needs to support log purge / retention policies

To delete the older WebJob log, I suggest you create a time triggered WebJob to delete the logs which you wanted.

Is there any AzureFunction code sample shows how to do the blob cleanup?

Code below is for your reference.

// Parse the connection string and return a reference to the storage account.
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(storageConnectionString);

// Create the table client.
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();

// Retrieve a reference to a container.
var container = blobClient.GetContainerReference("azure-webjobs-hosts");
// Query out all the blobs which created after 30 days
var blobs = container.GetDirectoryReference("output-logs").ListBlobs().OfType<CloudBlob>()
    .Where(b => b.Properties.LastModified < new DateTimeOffset(DateTime.Now.AddDays(-30)));
// Delete these blobs
foreach (var item in blobs)
{
    item.DeleteIfExists();
}