What is the best approach to append to a file in Elixir

Question

I've been stuck with some simple task. Let's assume we have some pseudo code:

Enum.each 1..1_000_000, fn(id) ->
    some_complex_method(id) |> save_results
end

Where save_results is

def save_results(data) do
    {:ok, file} = File.open "data.log", [:append]
    Enum.each(data, &(IO.binwrite(file, &1)))
    File.close file
end

The thing is if our range is really big we are spending time on opening and closing the file. How to make it handle open state and call close method when work is done?

Answer 1

Instead of :append as write mode use {:delayed_write, non_neg_integer, non_neg_integer} in order for your writes to be buffered until a certain amount of data to write is reached or a certain amount of time is up. Don't forget to use :append additionally if you only intend to append at the end of the file!

The first non_neg_integer is the Size in Bytes to buffer before a write occurs.

The second non_neg_integer is the Delay in Milliseconds to buffer before a write occurs. For a more detailed explanation visit the erlang documentation of :file.open/2. For other possible modes check out the elixir documentation for File.open/2

Your example could look like this:

Enum.each 1..1_000_000, fn(id) ->
    some_complex_method(id) |> save_results
end

Where save_results is

def save_results(data) do
    {:ok, file} = File.open "data.log", [:append, {:delayed_write, 100, 20}]
    Enum.each(data, &(IO.binwrite(file, &1)))
    File.close file
end

Whether this exact configuration is perfect for your case I cannot judge but it should definitely decrease the amount of File openings.

Translated the above configuration means:

Dear File module,
please buffer my writing to the file data.log until I asked you to write at least 100 Bytes of data or until you waited for 20 Milliseconds for me to give you more data to write.
Thank you very much!
Truly yours your elixir coder