Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Saving Powershell Object CSV String as a file to Azure Blob Storage

We are generating a Powershell object thru Azure Runbook and basically converting it into a CSV string.

We want to store this generated CSV String (generated from Azure Powershell Runbook) into Azure Blob Storage as a CSV File. Can someone help me how and whcih powershell command we can use to save this CSV String as File to Azure Blob Storage? I tried to look around and came across Push-OutputBindings function but not sure how I can use that one in Azure Powershell Runbook which module to import and not sure if it is part of Azure Functions V2 but any little basics on how I can use it will help me.

Thank you

like image 213
user42012 Avatar asked Oct 23 '19 13:10

user42012


People also ask

How do I get the connection string for Azure storage account using PowerShell?

To get a connection string with PowerShell, first get a StorageAccountContext object, then retrieve the ConnectionString property. To get a connection string with Azure CLI, call the az storage account show-connection-string command.

How do I run AzCopy in PowerShell?

Run AzCopy If you choose not to add the AzCopy directory to your path, you'll have to change directories to the location of your AzCopy executable and type azcopy or . \azcopy in Windows PowerShell command prompts. As an owner of your Azure Storage account, you aren't automatically assigned permissions to access data.

How to upload a CSV file to Azure Blob?

Every storage account in Azure has containers. Each container can contain blobs. We will be uploading the CSV file into the blob. Using the Azure storage will require obtaining the connection string to the Azure storage account. This is done as follows. Login to your Azure subscription. Go to your Azure storage account. Open Access Keys.

What do we know about Blob Storage in azure?

I will be using a slightly different approach: using the Azure Blob Storage API and C# to demonstrate this. What do we know about blob storage in Azure? Every storage account in Azure has containers. Each container can contain blobs. We will be uploading the CSV file into the blob.

How do I transfer files between local disk and Azure Blob Storage?

Use the Azure PowerShell module to create and manage Azure resources. Creating or managing Azure resources can be done from the PowerShell command line or in scripts. This guide describes using PowerShell to transfer files between local disk and Azure Blob storage. To access Azure Storage, you'll need an Azure subscription.

How to import CSV file to Azure Storage?

Solution 1: First you need to save the exported csv file in local system then you need to upload it to container in azure storage. I tried it in my system Solution 2: Upload the content directly to Azure storage by using sas Url.


1 Answers

Try the following code - with some modifications. Essentially idea is to store the CSV file locally and then, upload it to the blob storage(I tested it locally but not from the Runbook, but it should work there as well):

#file name will be a guid string to avoid overlapping     
$guid = New-Guid
$guidString =$guid.ToString()

# store csv string to random file with random name (guid) 
$LogFull = "$guidString.csv" 
$LogItem = New-Item -ItemType File -Name $LogFull

#ignore next two lines as you already have csv string
$Date = Get-Date
$csvstring = ConvertTo-Csv -InputObject $Date -Delimiter ';' -NoTypeInformation

#save csv string locally 
$csvstring | Out-File -FilePath $LogFull -Append

#and then upload it to blob storage

#Get key to storage account
$acctKey = (Get-AzureRmStorageAccountKey -Name storage_acc_name -ResourceGroupName EastUS-TestRG).Value[0]

#Map to the reports BLOB context
$storageContext = New-AzureStorageContext -StorageAccountName "StorageAccName" -StorageAccountKey "acc_key"

#Copy the file to the storage account
Set-AzureStorageBlobContent -File $LogFull -Container "your_container" -BlobType "Block" -Context $storageContext -Verbose

Alternative solution with Azure Functions

Try to have your Runbook calling your Http triggered Azure function, and pass the string as the parameter, or in the body. This is just a simple REST API call.

In Azure function, you can have Python, NodeJS or C# code that would put your string to CSV file in the blob store. There are plenty of tutorials for this topic, but first, you need to get your string to AF :)

Have a look at the example below, and try something similar(I have not tested it). Essentially the idea is to invoke simple REST API call and passing your payload in the request body:

[string]$Endpoint= "https://myfunction.azurewebsites.net/api/HttpTrigger?code=my_code_I_get_from_Azure_portal"
. . .
$payload =@" your payload "@
$Header = @{
   "Content-Type" = "text/plain";
}
$Result = Invoke-RestMethod -Uri $Endpoint -Method Post -body $Payload Headers $Header

Azure function URL+Code you get from Azure portal, if you click on Azure Function, you will see the button 'Get Function Url'.

like image 181
kgalic Avatar answered Oct 13 '22 21:10

kgalic