Uploading objects to google cloud storage buckets in c#

Question

Can someone please provide an example of how to use Google.Apis.Storage.v1 for uploading files to google cloud storage in c#?

Answer 1

I found that this basic operation is not as straight forward as you might expect. Google's documentation about it's Storage API is lacking in information about using it in C# (or any other .NET language). Searching for 'how to upload file to google cloud storage in c#' didn't exactly help me, so here is my working solution with some comments:

Preparation:

1. You need to create OAuth2 account in your Google Developers Console - go to Project/APIs & auth/Credentials.

2. Copy Client ID & Client Secret to your code. You will also need your Project name.

Code (it assumes that you've added Google.Apis.Storage.v1 via NuGet):

First, you need to authorize your requests:

var clientSecrets = new ClientSecrets();
clientSecrets.ClientId = clientId;
clientSecrets.ClientSecret = clientSecret;
//there are different scopes, which you can find here https://cloud.google.com/storage/docs/authentication
var scopes = new[] {@"https://www.googleapis.com/auth/devstorage.full_control"};

var cts = new CancellationTokenSource();
var userCredential = await GoogleWebAuthorizationBroker.AuthorizeAsync(clientSecrets,scopes, "yourGoogle@email", cts.Token);

Sometimes you might also want to refresh authorization token via:

await userCredential.RefreshTokenAsync(cts.Token);

You also need to create Storage Service:

var service = new Google.Apis.Storage.v1.StorageService();

Now you can make requests to Google Storage API. Let's start with creating a new bucket:

var newBucket = new Google.Apis.Storage.v1.Data.Bucket()
{
    Name = "your-bucket-name-1"
};

var newBucketQuery = service.Buckets.Insert(newBucket, projectName);
newBucketQuery.OauthToken = userCredential.Result.Token.AccessToken;
//you probably want to wrap this into try..catch block
newBucketQuery.Execute();

And it's done. Now, you can send a request to get list of all of your buckets:

var bucketsQuery = service.Buckets.List(projectName);
bucketsQuery.OauthToken = userCredential.Result.Token.AccessToken;
var buckets = bucketsQuery.Execute();

Last part is uploading new file:

//enter bucket name to which you want to upload file
var bucketToUpload = buckets.Items.FirstOrDefault().Name;
var newObject = new Object()
{
    Bucket = bucketToUpload,
    Name = "some-file-"+new Random().Next(1,666)
};

FileStream fileStream = null;
try
{
    var dir = Directory.GetCurrentDirectory();
    var path = Path.Combine(dir, "test.png");
    fileStream = new FileStream(path, FileMode.Open);
    var uploadRequest = new Google.Apis.Storage.v1.ObjectsResource.InsertMediaUpload(service, newObject,
    bucketToUpload,fileStream,"image/png");
    uploadRequest.OauthToken = userCredential.Result.Token.AccessToken;
    await uploadRequest.UploadAsync();
}
catch (Exception ex)
{
    Console.WriteLine(ex.Message);
}
finally
{
    if (fileStream != null)
    {
        fileStream.Dispose();
    }
}

And bam! New file will be visible in you Google Developers Console inside of selected bucket.

Answer 2

This is for Google.Cloud.Storage.V1 (not Google.Apis.Storage.v1), but appears to be a bit simpler to perform an upload now. I started with the Client libraries "Getting Started" instructions to create a service account and bucket, then experimented to find out how to upload an image.

The process I followed was:

1. Sign up for Google Cloud free trial
2. Create a new project in Google Cloud (remember the project name\ID for later)
3. Create a Project Owner service account - this will result in a json file being downloaded that contains the service account credentials. Remember where you put that file.
4. The getting started docs get you to add the path to the JSON credentials file into an environment variable called GOOGLE_APPLICATION_CREDENTIALS - I couldn't get this to work through the provided instructions. Turns out it is not required, as you can just read the JSON file into a string and pass it to the client constructor.
5. I created an empty WPF project as a starting point, and a single ViewModel to house the application logic.
6. Install the Google.Cloud.Storage.V1 nuget package and it should pull in all the dependencies it needs.

Onto the code.

MainWindow.xaml

<StackPanel>
    <Button
        Margin="50"
        Height="50"
        Content="BEGIN UPLOAD"
        Click="OnButtonClick" />
    <ContentControl
        Content="{Binding Path=ProgressBar}" />
</StackPanel>

MainWindow.xaml.cs

public partial class MainWindow
{
    readonly ViewModel _viewModel;

    public MainWindow()
    {
        _viewModel = new ViewModel(Dispatcher);
        DataContext = _viewModel;
        InitializeComponent();
    }

    void OnButtonClick(object sender, RoutedEventArgs args)
    {
        _viewModel.UploadAsync().ConfigureAwait(false);
    }
}

ViewModel.cs

public class ViewModel
{
    readonly Dispatcher _dispatcher;

    public ViewModel(Dispatcher dispatcher)
    {
        _dispatcher = dispatcher;
        ProgressBar = new ProgressBar {Height=30};
    }

    public async Task UploadAsync()
    {
        // Google Cloud Platform project ID.
        const string projectId = "project-id-goes-here";

        // The name for the new bucket.
        const string bucketName = projectId + "-test-bucket";

        // Path to the file to upload
        const string filePath = @"C:\path\to\image.jpg";

        var newObject = new Google.Apis.Storage.v1.Data.Object
        {
            Bucket = bucketName,
            Name = System.IO.Path.GetFileNameWithoutExtension(filePath),
            ContentType = "image/jpeg"
        };

        // read the JSON credential file saved when you created the service account
        var credential = Google.Apis.Auth.OAuth2.GoogleCredential.FromJson(System.IO.File.ReadAllText(
            @"c:\path\to\service-account-credentials.json"));

        // Instantiates a client.
        using (var storageClient = Google.Cloud.Storage.V1.StorageClient.Create(credential))
        {
            try
            {
                // Creates the new bucket. Only required the first time.
                // You can also create buckets through the GCP cloud console web interface
                storageClient.CreateBucket(projectId, bucketName);
                System.Windows.MessageBox.Show($"Bucket {bucketName} created.");

                // Open the image file filestream
                using (var fileStream = new System.IO.FileStream(filePath, System.IO.FileMode.Open))
                {
                    ProgressBar.Maximum = fileStream.Length;

                    // set minimum chunksize just to see progress updating
                    var uploadObjectOptions = new Google.Cloud.Storage.V1.UploadObjectOptions
                    {
                        ChunkSize = Google.Cloud.Storage.V1.UploadObjectOptions.MinimumChunkSize
                    };

                    // Hook up the progress callback
                    var progressReporter = new Progress<Google.Apis.Upload.IUploadProgress>(OnUploadProgress);

                    await storageClient.UploadObjectAsync(
                            newObject, 
                            fileStream,
                            uploadObjectOptions,
                            progress: progressReporter)
                        .ConfigureAwait(false);
                }

            }
            catch (Google.GoogleApiException e)
                when (e.Error.Code == 409)
            {
                // When creating the bucket - The bucket already exists.  That's fine.
                System.Windows.MessageBox.Show(e.Error.Message);
            }
            catch (Exception e)
            {
                // other exception
                System.Windows.MessageBox.Show(e.Message);
            }
        }
    }

    // Called when progress updates
    void OnUploadProgress(Google.Apis.Upload.IUploadProgress progress)
    {
        switch (progress.Status)
        {
            case Google.Apis.Upload.UploadStatus.Starting:
                ProgressBar.Minimum = 0;
                ProgressBar.Value = 0;

                break;
            case Google.Apis.Upload.UploadStatus.Completed:
                ProgressBar.Value = ProgressBar.Maximum;
                System.Windows.MessageBox.Show("Upload completed");

                break;
            case Google.Apis.Upload.UploadStatus.Uploading:
                UpdateProgressBar(progress.BytesSent);

                break;
            case Google.Apis.Upload.UploadStatus.Failed:
                System.Windows.MessageBox.Show("Upload failed"
                                               + Environment.NewLine
                                               + progress.Exception);
                break;
        }
    }

    void UpdateProgressBar(long value)
    {
        _dispatcher.Invoke(() => { ProgressBar.Value = value; });
    }

    // probably better to expose progress value directly and bind to 
    // a ProgressBar in the XAML
    public ProgressBar ProgressBar { get; }
}

Answer 3

You can use Google Cloud APIs without SDK in the following ways:

1. Required api-key.json file
2. Install package Google.Apis.Auth.OAuth2 in order to authorize the HTTP web request
3. You can set the default configuration for your application in this way
4. I did the same using .NET core web API and details are given below:

Url details:

"GoogleCloudStorageBaseUrl": "https://www.googleapis.com/upload/storage/v1/b/", "GoogleSpeechBaseUrl": "https://speech.googleapis.com/v1/operations/", "GoogleLongRunningRecognizeBaseUrl": "https://speech.googleapis.com/v1/speech:longrunningrecognize", "GoogleCloudScope": "https://www.googleapis.com/auth/cloud-platform",

public void GetConfiguration()
    {
        // Set global configuration
        bucketName = _configuration.GetValue<string>("BucketName");
        googleCloudStorageBaseUrl = _configuration.GetValue<string>("GoogleCloudStorageBaseUrl");
        googleSpeechBaseUrl = _configuration.GetValue<string>("GoogleSpeechBaseUrl");
        googleLongRunningRecognizeBaseUrl = _configuration.GetValue<string>("GoogleLongRunningRecognizeBaseUrl");

        // Set google cloud credentials
        string googleApplicationCredentialsPath = _configuration.GetValue<string>("GoogleCloudCredentialPath");
        using (Stream stream = new FileStream(googleApplicationCredentialsPath, FileMode.Open, FileAccess.Read))
            googleCredential = GoogleCredential.FromStream(stream).CreateScoped(_configuration.GetValue<string>("GoogleCloudScope"));

    }

Get Oauth token:

public string GetOAuthToken()
    {
        return googleCredential.UnderlyingCredential.GetAccessTokenForRequestAsync("https://accounts.google.com/o/oauth2/v2/auth", CancellationToken.None).Result;
    }

To upload file to cloud bucket:

public async Task<string> UploadMediaToCloud(string filePath, string objectName = null)
    {
        string bearerToken = GetOAuthToken();

        byte[] fileBytes = File.ReadAllBytes(filePath);
        objectName = objectName ?? Path.GetFileName(filePath);

        var baseUrl = new Uri(string.Format(googleCloudStorageBaseUrl + "" + bucketName + "/o?uploadType=media&name=" + objectName + ""));

        using (WebClient client = new WebClient())
        {
            client.Headers.Add(HttpRequestHeader.Authorization, "Bearer " + bearerToken);
            client.Headers.Add(HttpRequestHeader.ContentType, "application/octet-stream");

            byte[] response = await Task.Run(() => client.UploadData(baseUrl, "POST", fileBytes));
            string responseInString = Encoding.UTF8.GetString(response);
            return responseInString;
        }
    }

In order to perform any action to the cloud API, just need to make a HttpClient get/post request as per the requirement.

Thanks