S3 listObjects recursively node

Question

I want to recursively get all the files in an S3 bucket folder. I want to achieve the same behaviour as the ls -R command in Linux.

Here's an illustration

my_folder
  |_abc
  | |_abc_file.txt
  | |_xyz
  |   |_xyz_file.txt
  |
  |_file.txt

Considering the above directory structure. if I do

 const data = await s3.listObjectsV2({
    Prefix: 'my_folder/',
    Bucket: bucket,
    Delimiter: `/`,
}).promise();

Currently, I get the following data:

-CommonPrefixes: ["abc"]
-Contents:["file.txt"]

The expected behaviour is:

- Contents: ["abc/abc_file.txt", "abc/xyz/xyz_file.txt", "file.txt"]

I tried using ES6 generator functions and recursive functions to solve this problem and end up with a lot of messed up code.

I am using S3 node SDK

Answer 1

Amazon S3 is a flat object storage system that does not use directories. Instead, the Key (filename) of an object includes the full path of the object. Directories magically 'appear' based on the paths of existing objects, and can later disappear when there are no objects in that path.

A CommonPrefix is the Amazon S3-equivalent of showing a sub-directory.

When calling ListObjects() with a Delimiter parameter (eg Delimiter='/'), a list of subdirectories is returned in the CommonPrefixes field. This allows recursion through directories much like traditional storage systems. If you remove this parameter, all objects with the given Prefix will be returned, rather than just the 'current directory'.