Erlang File I/O: Large binary files and gzip streaming

Question

I have two questions regarding Erlang file i/o; what is the best way to achieve in Erlang:

1. reading large binary files (many gigabytes) without copying the whole file into memory
2. reading a gzipped binary file as a decompressed stream

Thanks!

Answer 1

1. See file:read/2 for sequential block access and file:pread/2,3 for random access.
2. See compressed option in file:open/2.

Answer 2

According to my experience, file:read/2 alone will be very slow if called frequently with small amounts of data, despite of read_ahead and raw. You must implement a binary buffer on top of that. If that is meant by block oriented processing then I agree.

I'm talking about runtimes of few hours (with file:read/2 only) vs. 2 minutes (with buffering implemented in pure Erlang).

Here are my measurements for reading a few 10 bytes at once:

%% Bufsize vs. runtime [ns]
%% 50       169369703
%% 100      118288832
%% 1000      70187233
%% 10000     64615506
%% 100000    65087411
%% 1000000   64747497

In this example, the performance doesn't really increase over 10 KB buffer size, because the relative overhead for file:read becomes small enough.