Unable to use createReadStream with Node 18 FormData

Question

Previously to Node 18 releasing fetch/FormData we could do:

import FormData from 'form-data'

const form = new FormData();
form.append('my_field', 'my value');
form.append('my_buffer', new Buffer(10));
form.append('my_file', fs.createReadStream('/foo/bar.jpg'));

However with the global FormData I can no longer pass a stream - the error at the "my_file" line is:

Argument of type 'ReadStream' is not assignable to parameter of type 'string | Blob'

I know this is still experimental so potentially a bug or is there an alternative way to do this - besides reading the entire file as a string...

Answer 1

Here is the implementation of pure streaming without reading the entire content in memory.

Node.JS built-in API:

import { open } from 'node:fs/promises';
import { File } from 'buffer';
  
const handle = await open('/path/to/your/file');

const stat = await handle.stat();

class MyFile extends File {
    // we should set correct size
    // otherwise we will encounter UND_ERR_REQ_CONTENT_LENGTH_MISMATCH
    size = stat.size;
}

const file = new MyFile([], 'file-name')


file.stream = function() {
    return handle.readableWebStream();
};

const formData = new FormData();

formData.append('file_key', file);

fetch('http://localhost', {
    method: 'post',
    body: formData
});

Using node-fetch:

import * as fs from 'fs';
import fetch, { FormData, File  } from 'node-fetch';

const stream = fs.createReadStream('/path/to/your/file');
const stat = fs.statSync('/path/to/your/file');

class MyFile extends File {
    size = stat.size;
}

const file = new MyFile([], 'file-name');

file.stream = function() {
    return stream;
};

const formData = new FormData();

formData.append('file_key', file);

fetch('http://localhost', {
    method: 'post',
    body: formData
});

Answer 2

Node.js 19.8+

You can use fs.openAsBlob.

import { openAsBlob } from 'node:fs/promises';
const blob = await openAsBlob('the.file.txt');

Then, you can append the file-backed blob to your form data.

const formData = new FormData();
formData.append('file', blob, name);

Node.js < 19.8

We were unable to find a CommonJS-compatible streaming Blob implementation, so we took inspiration from @roman-rodionov's answer and created a library that will create a streamable File. If using Node.js@<18.15 you'll also need to polyfill File.

You can install node-streamable-file to do this.

npm i node-streamable-file

We made a few tweaks to the implementation, most notably we removed the extra class definition. Here's our implementation if you'd like to do streaming file uploads without adding a dependency.

import { File } from 'buffer';
import { open } from 'node:fs/promises';
import { basename } from 'node:path';

export async function createStreamableFile(path: string): Promise<File> {
    const name = basename(path);
    const handle = await open(path);
    const { size } = await handle.stat();

    const file = new File([], name);
    file.stream = () => handle.readableWebStream();

    // Set correct size otherwise, fetch will encounter UND_ERR_REQ_CONTENT_LENGTH_MISMATCH
    Object.defineProperty(file, 'size', { get: () => size });

    return file;
}

Use createStreamableFile like so

const formData = new FormData();
const file = await createStreamableFile('path/to/file');
formData.append('file', file);

await fetch(url, {
  method: 'POST',
  body: formData
});

Additionally, if you're using TypeScript there's an issue with the File and Blob types. You can fix the problem by casting to the instance of File to unknown and then to Blob.