Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Google cloud function download file and redirect to bucket storage

I am trying to use a google cloud function in Node.js to download a file for wordpress repo then send the file into a google cloud bucket. I have the wordpress file downloading but it fails to write to the google bucket.

function writeToBucket(jsonObject){

/*
 *  Google API authentication
 */

var gcs = require('@google-cloud/storage')({
         projectId: 'wp-media-cdn',
         keyFilename: 'wp-media-cdn-d9d7c61bfad9.json'
});

/*
 *  rename image file with image size, format: size X size imgName
 */

var pluginUrl = "https://downloads.wordpress.org/plugin/bbpress.2.5.14.zip";
    newPluginName = "bbpress";

/*
 *  Read image into stream, upload image to bucket
 */

var request = require('request');
var fs = require('fs'); //used for createWriteString()

var myBucket = gcs.bucket('test_buckyy'); //PUT BUCKET NAME HERE
var file = myBucket.file(nnewPluginName);

// file.exists() returns true if file already in bucket, then returns file url, exits function
if(file.exists()){
    return 'https://storage.googleapis.com/${test_buckyy}/${file}';
}

//pipes image data into fileStream
var fileStream = myBucket.file(newImageName).createWriteStream();
request(imgUrl).pipe(fileStream)
    .on('error', function(err) {
        console.log('upload failed');
    })
    .on('finish', function() {
        console.log('file uploaded');
    });
/*
 *  return image url
 *  use getSignedUrl
 */


    return 'https://storage.googleapis.com/${test_buckyy}/${file}';

}
like image 749
Ryan Hubbard Avatar asked Nov 17 '17 21:11

Ryan Hubbard


People also ask

How do I upload data files to my cloud storage bucket?

In the Google Cloud console, go to the Cloud Storage Buckets page. In the list of buckets, click on the name of the bucket that you want to upload an object to. In the Objects tab for the bucket, either: Drag and drop the desired files from your desktop or file manager to the main pane in the Google Cloud console.

Can I upload files to Google Cloud Storage from URL?

Uploading files to Google Cloud Storage from a URL is possible, but there are a few things to keep in mind. First, you'll need to create a Google Cloud Storage bucket and give it a name. Next, you'll need to create a file object in the bucket and provide the URL of the file you want to upload.

How do I download directly to the cloud?

Offline download to cloudEnter the URL of the web file you want to save, and click on the cloud service you wish to upload web file to, for instance, Google Drive. Click on “Save to Cloud” to start to upload web file to cloud. Tips: When editing the file name, please enter a file name with its extension.


1 Answers

I just replicated your use case scenario and I successfully downloaded the file into the temporary folder of a Cloud Function and from there I copied this file into a bucket.

In order to achieve this, I downloaded the file using createWriteStream into the /tmp folder since is the only folder where we can store files in a Cloud Function, as stated in the Cloud Functions Execution Environment documentation.

After that, I just copied the file to a bucket following this Cloud Storage Uploading Objects documentation.

You can take a look of my sample function

Index.js

const {Storage} = require('@google-cloud/storage');

exports.writeToBucket = (req, res) => {
const http = require('http');
const fs = require('fs');

const file = fs.createWriteStream("/tmp/yourfile.jpg");
const request = http.get("YOUR_URL_TO_DOWNLOAD_A_FILE", function(response) {
  response.pipe(file);
});

console.log('file downloaded');

// Imports the Google Cloud client library
const {Storage} = require('@google-cloud/storage');

// Creates a client
const storage = new Storage();
const bucketName = 'YOUR_BUCKET_NAME';
const filename = '/tmp/yourfile.jpg';

// Uploads a local file to the bucket
storage.bucket(bucketName).upload(filename, {
  gzip: true,
  metadata: {
    cacheControl: 'no-cache',
  },
});

res.status(200).send(`${filename} uploaded to ${bucketName}.`);

};


package.json

{
  "name": "sample-http",
  "version": "0.0.1",
  "dependencies": {
        "@google-cloud/storage": "^3.0.3"
    }
}
like image 109
Chris32 Avatar answered Oct 30 '22 21:10

Chris32