How to take an element of a stream (Elixir) and keep the state of the stream?

Question

I love Elixir streams. In particular I've got a function which constructs an infinite stream of prime numbers; you can take the first 10,000, or all the primes below 1,000,000, or whatever, with the appropriate Stream/Enum operations (Enum.take(10_000) or Enum.take_while(& (&1 < 1_000_000)) respectively).

But suppose I don't know in advance how many primes I need. I get to a certain point and I say, hey, actually needed another thousand primes. Is there a way to say, get the first 10,000 elements of a stream, then save the resulting stream object somehow, so that I can get the next 1,000 on demand (and repeatedly of course)?

Answer 1

TL;DR Save the accumulator, not “Stream.”

The robust solution is provided by @Dogbert in comments: StreamSplit package seems to permorm exactly what was asked.

For the sake of history, my answer is: there are many Stream functions (all derived from Stream.transform/4, which is a generic stream implementation for nearly everything one might need) that might do the trick. For instance, consider Fibonacci numbers. One might implement them as:

stream = Stream.iterate({1, 1}, fn {acc, i} ->
  {acc + i, acc}
end)
#⇒ #Function<61.36862645/2 in Stream.unfold/2>
stream |> Enum.take(5) 
#⇒ [{1, 1}, {2, 1}, {3, 2}, {5, 3}, {8, 5}]
current = stream |> Enum.take(5) |> List.last
#⇒ {8, 5}

If you want to continue getting numbers:

#              ⇓⇓⇓⇓⇓⇓
Stream.iterate({8, 5}, fn {acc, i} ->
  {acc + i, acc}
end)

Just keep in an intermediate state and pass it as an initial value to stream function you are using to get primes. I personally do not see any advantage in keeping “tail” instead of the accumulator, but I might be definitely wrong.

Answer 2

You have a basic misunderstanding of Streams. A Stream is about creating composition of functions so that you can do complex processing on an enumerable with only one pass through the original enumerable.

It's easy to confuse a Stream with a Service, and with enough digging you can "pause" a Stream to create something service like. However, really what you want is a Prime Server. The minute you start thinking about "state" you should think about a GenServer.