Why is this Elixir code so slow?

Question

I'm currently very interested in learning Elixir and my typical approach of learning a new language is to build simple programs with it.

So I decided to write a (very simple) grep-like program (or module) which is shown here:

defmodule LineMatch do
  def file(s, path) do
    case File.open(path, [:read]) do
      {:ok, fd} -> match s, fd
      {_, error} -> IO.puts "#{:file.format_error error}"
    end
  end
  defp match(s, fd) do
    case IO.read fd, :line do
      {:error, error} -> IO.puts("oh noez! Error: #{error}")
      line -> match(s, line, fd)
    end
  end
  defp match(s, line, fd) when line !== :eof do
    if String.contains?(line, s) do
      IO.write line
    end
    match(s, IO.read(fd, :line), fd)
  end
  defp match(_, _line, _) when _line === :eof do
  end
end

This is most probably not the most elegant way to do it and I'm also quite new to functional programming, so I didn't expect it to be super fast. But it is not only not fast, it is actually super slow. So slow that I probably did some super obvious mistake.

Can anyone tell me, what it is and how to make it better?

I typically test the code with a seperate .exs file like

case System.argv do
  [searchTerm, path] -> LineMatch.file(searchTerm, path)
  _ -> IO.puts "Usage: lineMatch searchTerm path"
end

Answer 1

Rather than reading in the whole file as in lad2025's answer, you can get good performance by doing two things. First, use IO.binstream to build a stream of the file's lines, but as raw binary (for performance). Using IO.stream reads as UTF-8, and as such incurs extra cost to do the conversion as you are reading the file. If you need UTF-8 conversion, then it's going to be slow. Additionally, applying the filtering and mapping operations using the Stream.filter/2 and Stream.map/2 functions prevents you from iterating over the lines multiple times.

defmodule Grepper do
  def grep(path, str) do
    case File.open(path) do
      {:error, reason} -> IO.puts "Error grepping #{path}: #{reason}"
      {:ok, file} ->
        IO.binstream(file, :line)
        |> Stream.filter(&(String.contains?(&1, str)))
        |> Stream.map(&(IO.puts(IO.ANSI.green <> &1 <> IO.ANSI.reset)))
        |> Stream.run
    end
  end
end

I suspect the greatest issue with your code is the UTF-8 conversion, but it may be that by "pulling" from the file line by line by calling IO.read, rather than having the lines "pushed" to your filtering/printing operations using IO.stream|binstream, you are incurring some extra cost there. I'd have to look at Elixir's source to know for sure, but the above code was quite performant on my machine (I was searching a 143kb file from the Olson timezone database, not sure how it will perform on files of very large size as I don't have a good sample file handy).

Answer 2

Using of File.stream! will be much more efficient. Try it:

defmodule Grepper do
  def grep(path, str) do
    File.stream!(path)
      |> Stream.filter(&(String.contains?(&1, str)))
      |> Stream.map(&(IO.puts(IO.ANSI.green <> &1 <> IO.ANSI.reset)))
      |> Stream.run
  end
end