What is the best way to aggregate Streams into one DISTINCT with Java 8

Question

Suppose i have multiple java 8 streams that each stream potentially can be converted into Set<AppStory> , now I want with the best performance to aggregate all streams into one DISTINCT stream by ID , sorted by property ("lastUpdate")

There are several ways to do what but i want the fastest one , for example:

Set<AppStory> appStr1 =StreamSupport.stream(splititerato1, true).
map(storyId1 -> vertexToStory1(storyId1).collect(toSet());

Set<AppStory> appStr2 =StreamSupport.stream(splititerato2, true).
map(storyId2 -> vertexToStory2(storyId1).collect(toSet());

Set<AppStory> appStr3 =StreamSupport.stream(splititerato3, true).
map(storyId3 -> vertexToStory3(storyId3).collect(toSet());


Set<AppStory> set = new HashSet<>();
set.addAll(appStr1)
set.addAll(appStr2)
set.addAll(appStr3) , and than make sort by "lastUpdate"..

//POJO Object:
public class AppStory implements Comparable<AppStory> {
private String storyId;
private String ........... many other attributes......
public String getStoryId() {
    return storyId;
}
@Override
public int compareTo(AppStory o) {
    return this.getStoryId().compareTo(o.getStoryId());
   }
}

... but it is the old way.

How can I create ONE DISTINCT by ID sorted stream with BEST PERFORMANCE

somethink like :

  Set<AppStory> finalSet = distinctStream.sort((v1, v2) -> Integer.compare('not my issue').collect(toSet())

Any Ideas ?

BR

Vitaly

Answer 1

I think the parallel overhead is much greater than the actual work as you stated in the comments. So let your Streams do the job in sequential manner.

FYI: You should prefer using Stream::concat because slicing operations like Stream::limit can be bypassed by Stream::flatMap.

Stream::sorted is collecting every element in the Stream into a List, sort the List and then pushing the elements in the desired order down the pipeline. Then the elements are collected again. So this can be avoided by collecting the elements into a List and do the sorting afterwards. Using a List is a far better choice than using a Set because the order matters (I know there is a LinkedHashSet but you can't sort it).

This is the in my opinion the cleanest and maybe the fastest solution since we cannot prove it.

Stream<AppStory> appStr1 =StreamSupport.stream(splititerato1, false)
                                       .map(this::vertexToStory1);
Stream<AppStory> appStr2 =StreamSupport.stream(splititerato2, false)
                                       .map(this::vertexToStory2);
Stream<AppStory> appStr3 =StreamSupport.stream(splititerato3, false)
                                       .map(this::vertexToStory3);

List<AppStory> stories = Stream.concat(Stream.concat(appStr1, appStr2), appStr3)
                               .distinct().collect(Collectors.toList());
// assuming AppStory::getLastUpdateTime is of type `long`
stories.sort(Comparator.comparingLong(AppStory::getLastUpdateTime));