Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

async upload multiple files to google cloud storage bucket

I'm trying to upload multiple files to a Google Cloud Storage bucket using NodeJS. I want all files to be uploaded before continuing. I tried several approaches but I can't seem to get it right.

const jpegImages = await fs.readdir(jpegFolder);

console.log('start uploading');

await jpegImages.forEach(async fileName => {
    await bucket.upload(
        path.join(jpegFolder, fileName),
        {destination: fileName}
     ).then( () => {
         console.log(fileName + ' uploaded');
     })
})

console.log('finished uploading');

This gives me the following output, which is not what I expect. Why is the 'finished uploading' log not executed after uploading the files?

start uploading
finished uploading
image1.jpeg uploaded
image2.jpeg uploaded
image3.jpeg uploaded
like image 395
Kirk Olson Avatar asked Jan 08 '19 15:01

Kirk Olson


People also ask

Can I upload files to Google Cloud Storage from URL?

Uploading files to Google Cloud Storage from a URL is possible, but there are a few things to keep in mind. First, you'll need to create a Google Cloud Storage bucket and give it a name. Next, you'll need to create a file object in the bucket and provide the URL of the file you want to upload.

How do I upload a bucket file to GCP?

In the Google Cloud console, go to the Cloud Storage Buckets page. In the list of buckets, click on the name of the bucket that you want to upload an object to. In the Objects tab for the bucket, either: Drag and drop the desired files from your desktop or file manager to the main pane in the Google Cloud console.

How do I import a CSV file into Google Cloud Storage?

Find your app and select your app options caret ( ^ ). Select All Settings > Raw Data Export > CSV Upload. Select Google Cloud Storage from the dropdown menu. Upload your Service Account Key credential file.


1 Answers

async/await doesn't work with forEach and other array methods.

If you don't need sequential uploading (files can be uploaded in parallel) you could create an array of Promises and use Promise.all() to execute them all at once.

const jpegImages = await fs.readdir(jpegFolder);

console.log('start uploading');

await Promise
    .all(jpegImages.map(fileName => {
        return bucket.upload(path.join(jpegFolder, fileName), {destination: fileName})
    }))
    .then(() => {
        console.log('All images uploaded')
    })
    .catch(error => {
        console.error(`Error occured during images uploading: ${error}`);
    });

console.log('finished uploading');
like image 178
Ihor Sakailiuk Avatar answered Sep 28 '22 01:09

Ihor Sakailiuk