Creating a zip file in situ within azure blob storage

Question

I have files stored in one container within a blob storage account. I need to create a zip file in a second container containing the files from the first container.

I have a solution that works using a worker role and DotNetZip but because the zip file could end up being 1GB in size I am concerned that doing all the work in-process, using MemoryStream objects etc. is not the best way of doing this. My biggest concern is that of memory usage and freeing up resources given that this process could happen several times a day.

Below is some very stripped down code showing the basic process in the worker role:

using (ZipFile zipFile = new ZipFile())
{
    foreach (var uri in uriCollection)
    {
        var blob = new CloudBlob(uri);

        byte[] fileBytes = blob.DownloadByteArray();

        using (var fileStream = new MemoryStream(fileBytes))
        {
            fileStream.Seek(0, SeekOrigin.Begin);

            byte[] bytes = CryptoHelp.EncryptAsBytes(fileStream, "password", null);

            zipFile.AddEntry("entry name", bytes);
        }
    }

    using (var zipStream = new MemoryStream())
    {
        zipFile.Save(zipStream);
        zipStream.Seek(0, SeekOrigin.Begin);

        var blobRef = ContainerDirectory.GetBlobReference("output uri");
        blobRef.UploadFromStream(zipStream);
    }

}

Can someone suggest a better approach please?

Answer 1

At the time of writing this question, I was unaware of the LocalStorage options available in Azure. I was able to write files individually to this and the work with them within the LocalStorage and then write them back to blob storage.

Answer 2

If all you are worried aobut is your memorysteam taking up too much memory then what you can do is implement your own stream and as your stream is being read, you you add your zip files to the stream and remove already read files from the stream. This will keep your memory stream size to the size of one file.