How to efficiently initialize huge sparse arrays in Julia?

Question

There are two ways one can initialize a NXN sparse matrix, whose entries are to be read from one/multiple text files. Which one is faster? I need the more efficient one, as N is large, typically 10^6.

1). I could store the (x,y) indices in arrays x, y, the entries in an array v and declare

K = sparse(x,y,value);

2). I could declare K = spzeros(N)

then read of the (i,j) coordinates and values v and insert them as K[i,j]=v;

as they are being read. I found no tips about this on Julia’s page on sparse arrays.

Answer 1

Don’t insert values one by one: that will be tremendously inefficient since the storage in the sparse matrix needs to be reallocated over and over again.

You can also use BenchmarkTools.jl to verify this:

julia> using SparseArrays

julia> using BenchmarkTools

julia> I = rand(1:1000, 1000); J = rand(1:1000, 1000); X = rand(1000);

julia> function fill_spzeros(I, J, X)
         x = spzeros(1000, 1000)
         @assert axes(I) == axes(J) == axes(X)
         @inbounds for i in eachindex(I)
           x[I[i], J[i]] = X[i]
         end
         x
       end
fill_spzeros (generic function with 1 method)

julia> @btime sparse($I, $J, $X);
  10.713 μs (12 allocations: 55.80 KiB)

julia> @btime fill_spzeros($I, $J, $X);
  96.068 μs (22 allocations: 40.83 KiB)

Original post can be found here