Solution for retrieving large files from database with NestJS

Question

My NestJS app was having issues with uploading large files. I used bodyParser to set the limit to 50 MB and it workws.

However, the issue started again after I uploaded many files. I have tried to set it to 1 GB 500 MB and the parameterLimit from 50000 to 500000 but the issue persists.

I can still upload files but I can’t fetch them. And that should be normal since I'm not uploading many files for one Post request. To fetch it all, I have to query all of it at the same time to an array of data, so the requested data becomes large and that is where NestJS throws an error. I don’t know where to set the outgoing data size limits.

Here is my controllerL

@Get()
    async fetchAll(): Promise<Intro[]> {
    return await this.sejourService.findAll();
}

and here is my service:

async findAll(): Promise <Intro[]> {
    return await this.introModel.find();
 }

and in my main:

app.use(bodyParser.json({limi: '900mb'}));
app.use(bodyparser.urlencoded({ limit: '900mb', extended: true})

Answer 1

I'm not sure what issues you were running into with the FileInterceptor, but it can absolutely handle gig+ size files. A simple endpoint like

@Controller()
export class AppController {

  @Post('file')
  @UseInterceptors(FileInterceptor('file'))
  uploadFile(@UploadedFile() file: Express.Multer.File) {
    console.log(file);
    return { status: 'ok' };
  }
}

is all that's needed to accept the incoming file. I tested this using curl like

 curl http://localhost:3000/file -F 'file=@~/Downloads/1GB.zip'
{"status":"ok"}

And got the following log in my server

{
  fieldname: 'file',
  originalname: '1GB.zip',
  encoding: '7bit',
  mimetype: 'application/octet-stream',
  buffer: <Buffer ff da 18 9f 40 8d 04 a1 31 5f 6c 15 62 8b 6c 56 4f e7 84 d8 f3 01 4a 78 b7 36 9e ff b9 a8 f8 a7 06 f9 5a 89 57 a3 39 0f 0b 90 dc 30 8f 85 46 b1 ce 41 ... 1073741774 more bytes>,
  size: 1073741824
}

As you can see, the size is there, the mimetype is set, and the buffer property now has all the data encoded in the file.

---

I was not able to create a string of such size to send through curl or postman without killing the process running the applications/commands, but if for whatever reason the above is not doable for you, the following should be fine in the main.ts

import { NestFactory } from '@nestjs/core';
import * as bodyParser from 'body-parser';
import { AppModule } from './app.module';

async function bootstrap() {
  const app = await NestFactory.create(AppModule, { bodyParser: false });
  app.use(bodyParser.urlencoded({ limit: '2gb', extended: true }));
  app.use(bodyParser.json({ limit: '2gb' });
  await app.listen(3000);
}
bootstrap();

---

To retrieve the large file, all that's needed is something like the StreamableFile API from Nest and returning the read stream like so:

 @Get(':file')
  getFile(@Param('file') fileName) {
    return new StreamableFile(
      createReadStream(join(process.cwd(), 'uploads', fileName)),
    );
  }

This assumes you've uploaded the files to ./uploads and you have the name available. Tested, and it works fine.