nodejs crypto module, does hash.update() store all input in memory

Question

I have an API route that proxies a file upload from the browser/client to AWS S3.

This API route attempts to stream the file as it is uploaded to avoid buffering the entire contents of the file in memory on the server.

However, the route also attempts to calculate an MD5 checksum of the file's body. As each part of the file is chunked, the hash.update() method is invoked w/ the chunk.

http://nodejs.org/api/crypto.html#crypto_hash_update_data_input_encoding

var crypto = require('crypto');
var hash = crypto.createHash('md5');
function write (chunk) {
  // invoked many times as file is uploaded
  hash.update(chunk);
}
function done() {
  // will hash buffer all chunks in memory at this point?
  hash.digest('hex');
}

Will the instance of Hash buffer all the contents of the file in order to perform the hash calculation (thus defeating the goal of avoiding buffering the entire file's contents in memory)? Or can an MD5 hash be calculated incrementally, without ever having the entire input available to perform the calculation?

Answer 1

MD5 and some other hash functions are based on the Merkle–Damgård construction. It supports the incremental/progressive/streaming hashing of data. After the data is transformed into an internal state (which has a fixed size) a last finalization step is performed to generate the final hash by padding and processing the last block and afterwards by simply returning the final state.

This is probably also why many hashing library functions are designed in such a way with an update and a finalization step.

To answer your question: No, the file content is not kept in a buffer, but is rather transformed into a fixed size internal state.