Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Upload files and folder into Azure Blob Storage

I have created a storage account in Azure and created a container. I am trying to upload files stored in my Server the files are stored within 800 folders. I have tried doing this with this Powershell script however it does not work with the subfolders.

$LocalFolder = "\\Server\Data" 

Add-AzureAccount 


# Set a default Azure subscription.
Select-AzureSubscription -SubscriptionName 'nameofsubscription' –Default


$Key = Get-AzureStorageKey -StorageAccountName  mydatastorename

$Context = New-AzureStorageContext -StorageAccountKey $Key.Primary -StorageAccountName mydatastorename

foreach ($folder in Get-ChildItem $LocalFolder)
    {

            ls –Recurse –Path $LocalFolder |Set-AzureStorageBlobContent  -Container nameofcontainer -Context $Context -BlobType Block
    }

If set the $LocalFolder as "\Server\Data\subfolders001" the files in subfolder001 get uploaded to the container. But when I keep it as "\Server\Data" then it does not work.

I want the script to upload all the sub folders and files within into the storage container.

I have added the output I get when I run it

I don't get any error message, but one warning message each subfolder

 WARNING: Can not upload the directory '\\Server\Data\subfolders001' to azure. If you want to upload directory, please use "ls -File -Recurse | Set-AzureStorageBlobContent -Container containerName".

Then noting happens after waiting for a while I have to stop the powershell script.

like image 541
Vaishak Shanbhag Avatar asked Jul 13 '16 14:07

Vaishak Shanbhag


People also ask

How do I upload folders to azure blob storage?

Upload a folder to a blob containerOn the main pane's toolbar, select Upload, and then Upload Folder from the drop-down menu.

Can we upload folder to blob storage?

You can upload files and directories to Blob storage by using the AzCopy v10 command-line utility.

How do I transfer files to blob storage?

Click on the ... to the right of the Files box, select one or multiple files to upload from the file system and click Upload to begin uploading the files. To download data, selecting the blob in the corresponding container to download and click Download.


1 Answers

Was just solving this myself, see my solution below:

$StorageAccountName = "storageAccountName"
$StorageAccountKey = "StorageAccountKey"
$ContainerName = "ContainerName"
$sourceFileRootDirectory = "AbsolutePathToStartingDirectory" # i.e. D:\Docs

function Upload-FileToAzureStorageContainer {
    [cmdletbinding()]
    param(
        $StorageAccountName,
        $StorageAccountKey,
        $ContainerName,
        $sourceFileRootDirectory,
        $Force
    )

    $ctx = New-AzureStorageContext -StorageAccountName $StorageAccountName -StorageAccountKey $StorageAccountKey
    $container = Get-AzureStorageContainer -Name $ContainerName -Context $ctx

    $container.CloudBlobContainer.Uri.AbsoluteUri
    if ($container) {
        $filesToUpload = Get-ChildItem $sourceFileRootDirectory -Recurse -File

        foreach ($x in $filesToUpload) {
            $targetPath = ($x.fullname.Substring($sourceFileRootDirectory.Length + 1)).Replace("\", "/")

            Write-Verbose "Uploading $("\" + $x.fullname.Substring($sourceFileRootDirectory.Length + 1)) to $($container.CloudBlobContainer.Uri.AbsoluteUri + "/" + $targetPath)"
            Set-AzureStorageBlobContent -File $x.fullname -Container $container.Name -Blob $targetPath -Context $ctx -Force:$Force | Out-Null
        }
    }
}

Running:

Upload-FileToAzureStorageContainer -StorageAccountName $StorageAccountName -StorageAccountKey $StorageAccountKey -ContainerName $ContainerName -sourceFileRootDirectory $sourceFileRootDirectory -Verbose

You should see that it prints the following output as it runs:

VERBOSE: Uploading testfile.json to https://storagecontainername.blob.core.windows.net/path/testfile.json
VERBOSE: Performing the operation "Set" on target "/path/testfile.json".
ICloudBlob        : Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob
BlobType          : BlockBlob
Length            : 0
ContentType       : application/octet-stream
LastModified      : 23/03/2017 14:20:53 +00:00
SnapshotTime      :
ContinuationToken :
Context           : Microsoft.WindowsAzure.Commands.Common.Storage.AzureStorageContext
Name              : /path/testfile.json

VERBOSE: Transfer Summary
--------------------------------
Total:  1.
Successful: 1.
Failed: 0.

I admit this isn't perfect but with a bit of effort you could extend it to cater for more complex scenario file uploads but it should do the trick if you wanted to upload the contents of a directory to a target storage container.

Azure Storage Containers are essentially directories that can't contain other storage containers. So in order to implement a folder structure in an Azure Storage Container, you need to prefix the file with the path up to the root folder you started searching from. I.e. If your root is :

D:\Files

And Files contains:

Folder 1
  -> File 1.txt
File 2.txt

You need to set the target paths to be

$StorageContainerObject.CloudBlobContainer.Uri.AbsoluteUri + "\Files\Folder 1\File 1.txt"
$StorageContainerObject.CloudBlobContainer.Uri.AbsoluteUri + "\Files\File 2.txt"

If you view your storage container using a tool like Azure Storage Explorer then it recognises the file structures even though the files are all stored within one container.

Thanks,

Jamie

like image 77
Jamie Avatar answered Sep 28 '22 06:09

Jamie