Java 8 Stream distinct is not working

Question

This is what i am doing:

List scores = Stream.concat(oldEntries.stream(), newEntries.stream())
                    .sorted()
                    .distinct()
                    .limit(maxSize)
                    .collect(Collectors.toList());

I am expecting a sorted list without any duplicates, but some times there is duplicate in the list.

I have override the hashCode and equals method, I have also observed that these methods are returning the correct value every time. Can any one see what is wrong with my stream?

This is my equals() and hashCode() They are auto generated by IDEA :

..
private int userId;
private int levelId;
private int score;

@Override
public boolean equals(Object o) {

    if (this == o) return true;
    if (o == null || getClass() != o.getClass()) return false;

    Score score = (Score) o;

    if (userId != score.userId) return false;
    return levelId == score.levelId;

}

@Override
public int hashCode() {
    int result = userId;
    result = 31 * result + levelId;
    return result;
}

public int compareTo(Score other) {

    if (other == null) {
        return 1;
    } else {
        return Integer.compare(other.score, this.score);
    }
}

 ..

Answer 1

Your stream is first ordered according to compareTo, i.e. using score.

It's then "distinctified" using equals(), i.e. using userId and levelId. According to the javadoc:

For ordered streams, the selection of distinct elements is stable (for duplicated elements, the element appearing first in the encounter order is preserved.) For unordered streams, no stability guarantees are made.

Example:

score 1, user 2, level 3
score 3, user 2, level 3
score 1, user 3, level 1

After sorting...

score 1, user 2, level 3
score 1, user 3, level 1
score 3, user 2, level 3

Distinct now does nothing, because the elements are not equal according to user/level. This can result in "duplicate" elements, because you're ordering the stream based on one thing, but determining equality by an entirely different thing.