how to stream read directory in node.js?

Question

Suppose I have a directory that contains 100K+ or even 500k+ files. I want to read the directory with fs.readdir, but it's async not stream. Someone tell me that async use memory before done read the entire file list.

So what is the solution? I want to readdir with stream approach. Can I?

Answer 1

Now there is a way to do it with async iteration! You can do:

const dir = fs.opendirSync('/tmp')

for await (let file of dir) {
  console.log(file.name)
}

To turn it into a stream:

const _pipeline = util.promisify(pipeline)
await _pipeline([
  Readable.from(dir),
  ... // consume!
])