Efficient Sequence conversion to String

Question

I have a string sequence Seq[String] which represents stdin input lines.

Those lines map to a model entity, but it is not guaranteed that 1 line = 1 entity instance.

Each entity is delimited with a special string that will not occur anywhere else in the input.

My solution was something like:

val entities = lines.mkString.split(myDelimiter).map(parseEntity)

parseEntity implementation is not relevant, it gets a String and maps to a case class which represents the model entity

The problem is with a given input, I get an OutOfMemoryException on the lines.mkString. Would a fold/foldLeft/foldRight be more efficient? Or do you have any better alternative?

Answer 1

You can solve this using akka streams and delimiter framing. See this section of the documentation for the basic approach.

import akka.actor.ActorSystem
import akka.stream.ActorMaterializer
import akka.stream.scaladsl.{Framing, Source}
import akka.util.ByteString

val example = (0 until 100).mkString("delimiter").grouped(8).toIndexedSeq
val framing = Framing.delimiter(ByteString("delimiter"), 1000)

implicit val system = ActorSystem()

implicit val mat = ActorMaterializer()

Source(example)
  .map(ByteString.apply)
  .via(framing)
  .map(_.utf8String)
  .runForeach(println)

The conversion to and from ByteString is a bit annoying, but Framing.delimiter is only defined for ByteString.

If you are fine with a more pure functional approach, fs2 will also offer primitives to solve this problem.