Menu
I've got a remote server with a bunch of static files (outside of Azure). I've created a worker role in Azure and I want to use the worker role to transfer these files using SFTP from my remote server straight to my blob storage account (without copying locally to the worker role). Is there an established workflow/best practices on how to do this?
The closest thing I was able to find was this question: Copy file from URL to Azure BLOB
However to use StartCopyFromBlob I would need to have a publicly accessible URL which is not the case.
Also some of these files may be >100mb or >500mb, should that raise any problems?
Thanks.
794
To get started, enable SFTP support, create a local user, and assign permissions for that local user. Then, you can use any SFTP client to securely connect and then transfer files. For step-by-step guidance, see Connect to Azure Blob Storage by using the SSH File Transfer Protocol (SFTP).
You can securely connect to the Blob Storage endpoint of an Azure Storage account by using an SFTP client, and then upload and download files. This article shows you how to enable SFTP, and then connect to Blob Storage by using an SFTP client.
Upload a directory by using the azcopy copy command. This example copies a directory (and all of the files in that directory) to a blob container. The result is a directory in the container by the same name. This example encloses path arguments with single quotes ('').
The simplest approach would be to install the Azure CLI directly on the remote server. You can then use the Azure CLI to transfer these files directly to Azure Blob Storage. No need for SFTP (Azure Blob Storage doesn't expose an SFTP interface) or worker roles (your remote server isn't serving files over a web interface).
If this is not an option, the other approach would be to do what @Mark Volders suggested, and provision an SFTP server on Azure. You can then push files from the remote server to the SFTP server. The SFTP server would then push the file to Azure Blob Storage using the Azure CLI, and delete the local file upon success.
For the SFTP server, one hurdle is making sure files are copied to Azure Blob Storage as soon as the SFTP client is done transferring a file. A common approach is to use Incron, which is a service that listens for file events (in this case, the IN_CLOSE_WRITE event). There's a product SFTP Gateway on the Azure Marketplace that does all this (disclosure: I'm one of the developers for this product) so you don't need to spend time implementing this from scratch.
Also, file sizes of >500mb should not be a problem for either the Azure CLI or SFTP.
62
You might want to tackle this the other way around, by setting up an FTP server on Azure which save the files directly to Azure storage.
A good explanation on how to do this by use of the Preview Files can be found on http://fabriccontroller.net/blog/posts/deploying-a-load-balanced-high-available-ftp-server-with-azure-files/
3
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
