Julia: CSV.write very memory inefficient?

Question

I noticed that when saving large dataframes as CSVs the memory allocations are an order of magnitude higher than the size of the dataframe in memory (or the size of the CSV file on disk), at least by a factor of 10. Why is this the case? And is there a way to prevent this? Ie is there a way to save a dataframe to disk without using (much) more memory than the actual dataframe?

In the example below I generate a dataframe with one integer column and 10m rows. It weighs 76MB but writing the CSV allocates 1.35GB.

using DataFrames, CSV

function generate_df(n::Int64)
    DataFrame!(a = 1:n)
end

julia> @time tmp = generate_df2(10000000);
  0.671053 seconds (2.45 M allocations: 199.961 MiB)

julia> Base.summarysize(tmp) / 1024 / 1024
76.29454803466797

julia> @time CSV.write("~/tmp/test.csv", tmp)
  3.199506 seconds (60.11 M allocations: 1.351 GiB)

Answer 1

What you see is not related to CSV.write, but to the fact that DataFrame is type-unstable. This means that it will allocate when iterating rows and accessing their contents. Here is an example:

julia> df = DataFrame(a=1:10000000);

julia> f(x) = x[1]
f (generic function with 1 method)

julia> @time sum(f, eachrow(df)) # after compilation
  0.960045 seconds (40.07 M allocations: 613.918 MiB, 4.18% gc time)
50000005000000

This is a deliberate design decision to avoid unacceptable compilation times for very wide data frames (which are common in practice in certain fields of application). Now, this is the way to reduce allocations:

julia> @time CSV.write("test.csv", df) # after compilation
  1.976654 seconds (60.00 M allocations: 1.345 GiB, 5.64% gc time)
"test.csv"

julia> @time CSV.write("test.csv", Tables.columntable(df)) # after compilation
  0.439597 seconds (36 allocations: 4.002 MiB)
"test.csv"

(this will work OK if the table is narrow, for wide tables it might hit compilation time issues)

This is one of the patterns that are often encountered in Julia (even Julia itself works this way as args field in Expr is Vector{Any}): often you are OK with type unstable code if you do not care about performance (but want to avoid excessive compilation latency), and it is easy to switch to type-stable mode where compilation time does not matter and type-stability does.

Answer 2

Python Pandas:

import pandas as pd
 df = pd.DataFrame({'a': range(10_000_000)})
 %time  df.to_csv("test_py.csv", index=False)

memory consumption (measured in Task Manager): 135 MB (before writing) -> 151 MB (during writing), Wall time: 8.39 s

Julia:

using DataFrames, CSV
df = DataFrame(a=1:10_000_000)
@time CSV.write("test_jl.csv", df)
@time CSV.write("test_jl.csv", df)

memory consumption: 284 MB (before writing) -> 332 MB (after 1st writing), 2.196639 seconds (51.42 M allocations: 1.270 GiB, 7.49% gc time)

2nd execution (no compilation required anymore): -> 357 MB, 1.701374 seconds (50.00 M allocations: 1.196 GiB, 6.26% gc time)

The memory increase of Python and Julia during writing of the CSV file is similar (~ 15 MB). In Julia, I observe a significant memory increase after execution of the first write command, probably due to cashing of compiled code.

Note that allocations != memory requirement. Even though 1.2 GB memory is allocated in total, only 15 MB is used at the same time (max value).

Regarding performance, Julia is nearly 4 times faster than Python, even including compilation time.