Finite generated Stream in Java - how to create one?

Question

In Java, one can easily generate an infinite stream with Stream.generate(supplier). However, I would need to generate a stream that will eventually finish.

Imagine, for example, I want a stream of all files in a directory. The number of files can be huge, therefore I can not gather all the data upfront and create a stream from them (via collection.stream()). I need to generate the sequence piece by piece. But the stream will obviously finish at some point, and terminal operators like (collect() or findAny()) need to work on it, so Stream.generate(supplier) is not suitable here.

Is there any reasonable easy way to do this in Java, without implementing the entire Stream interface on my own?

I can think of a simple hack - doing it with infinite Stream.generate(supplier), and providing null or throwing an exception when all the actual values are taken. But it would break the standard stream operators, I could use it only with my own operators that are aware of this behaviour.

CLARIFICATION

People in the comments are proposing me takeWhile() operator. This is not what I meant. How to phrase the question better... I am not asking how to filter (or limit) an existing stream, I am asking how to create (generate) the stream - dynamically, without loading all the elements upfront, but the stream would have a finite size (unknown in advance).

SOLUTION

The code I was looking for is

    Iterator it = myCustomIteratorThatGeneratesTheSequence();     StreamSupport.stream(Spliterators.spliteratorUnknownSize(it, Spliterator.DISTINCT), false); 

I just looked into java.nio.file.Files, how the list(path) method is implemented.

Answer 1

Is there any reasonable easy way to do this in Java, without implementing the entire Stream interface on my own?

A simple .limit() guarantees that it will terminate. But that's not always powerful enough.

After the Stream factory methods the simplest approach for creating customs stream sources without reimplementing the stream processing pipeline is subclassing java.util.Spliterators.AbstractSpliterator<T> and passing it to java.util.stream.StreamSupport.stream(Supplier<? extends Spliterator<T>>, int, boolean)

If you're intending to use parallel streams note that AbstractSpliterator only yields suboptimal splitting. If you have more control over your source fully implementing the Spliterator interface can better.

For example, the following snippet would create a Stream providing an infinite sequence 1,2,3...

in that particular example you could use IntStream.range()

But the stream will obviously finish at some point, and terminal operators like (collect() or findAny()) need to work on it.

short-circuiting operations like findAny() can actually finish on an infinite stream, as long as there is any element that matches.

Java 9 introduces Stream.iterate to generate finite streams for some simple cases.

Answer 2

Kotlin code to create Stream of JsonNode from InputStream

     private fun InputStream.toJsonNodeStream(): Stream<JsonNode> {         return StreamSupport.stream(                 Spliterators.spliteratorUnknownSize(this.toJsonNodeIterator(), Spliterator.ORDERED),                 false         )     }      private fun InputStream.toJsonNodeIterator(): Iterator<JsonNode> {         val jsonParser = objectMapper.factory.createParser(this)          return object: Iterator<JsonNode> {              override fun hasNext(): Boolean {                 var token = jsonParser.nextToken()                 while (token != null) {                     if (token == JsonToken.START_OBJECT) {                         return true                     }                     token = jsonParser.nextToken()                 }                 return false             }              override fun next(): JsonNode {                 return jsonParser.readValueAsTree()             }         }     }