AWS Lambda to list S3 bucket objects and subdirectories inside with file versioning

Question

I'm new to lambda and trying to list S3 bucket objects that has nested subdirectories , here is the structure:

- mybucket/folder1/project1/samplev1.txt
- mybucket/folder1/project1/sampleVer2.txt
- mybucket/folder2/
- mybucket/folder3/

here is my lambda code:

const AWS = require('aws-sdk');
const s3 = new AWS.S3();

exports.handler = async (event) => {
  const allKeys = [];
  await getKeys({ Bucket: 'ru-mybucket' }, allKeys);
  console.log(allKeys)
  return allKeys;
};

async function getKeys(params, keys){
  const response = await s3.listObjectsV2(params).promise();
  response.Contents.forEach(obj => keys.push(obj.Key));

  if (response.IsTruncated) {
    const newParams = Object.assign({}, params);
    newParams.ContinuationToken = response.NextContinuationToken;
    await getKeys(newParams, keys); // RECURSIVE CALL
  }
}

The function list all objects keys inside my bucket with their nested subdirectories and files, The problem is how to list nested objects including their version, how can I achieve this? where to add (listObjectVersions) in my code to able loop over the objects which has versions?

Answer 1

I have done this code for you, in this example you can be seen the unit test and its implementation (apigateway).

Basically, i retrieved all keys from my bucket and then iterate them. Finally i got all objects, and i query s3.listObjectsV2 passing two parameter the bucket and key.

Below a pseudocode:

s3.listObjectsV2({ Bucket: your_bucket, MaxKeys: 1000 }).forEach( element => {
   s3.listObjectVersions({ Bucket: your_bucket, Prefix: element.Key})
})

For that you can run the code sample, you must run the following command:

npm install
npm run deploy

For that you can test the code:

npm run test

Don't forget put your bucket:

./package.json
./tests/test.js

Important: i used serverless framework for this solution.