How can I download big files in Deno?

Question

I'm trying to download a 10GB file, but only 4GB get saved to disk, and memory is growing a lot.

const res = await fetch('https://speed.hetzner.de/10GB.bin');
const file = await Deno.open('./10gb.bin', { create: true, write: true })

const ab = new Uint8Array(await res.arrayBuffer())
await Deno.writeAll(file, ab)

Answer 1

You're buffering the response, that's why the memory is growing.

You can iterate through res.body since it's currently a ReadableStream which implements Symbol.asyncIterator and use Deno.writeAll on each chunk.

for await(const chunk of res.body) {
    await Deno.writeAll(file, chunk);
}
file.close();

---

You can also use fromStreamReader from std/io (>= [email protected]) to convert res.body to a Reader that can be used in Deno.copy

import { fromStreamReader } from "https://deno.land/[email protected]/io/streams.ts";
const res = await fetch('https://speed.hetzner.de/10GB.bin');
const file = await Deno.open('./10gb.bin', { create: true, write: true })

const reader = fromStreamReader(res.body!.getReader());
await Deno.copy(reader, file);
file.close();

---

Regarding why it stops at 4GB I'm not sure, but it may have to do with ArrayBuffer / UInt8Array limits, since 4GB is around 2³² bytes, which is the limit of TypedArray, at least in most runtimes.