Why is findFirst() throwing a NullPointerException if I'm priorly filtering only for present() values?

Question

I have a Stream of Strings, and am mapping each String to Optional<String>. Since I'm filtering empty Optionals afterwards, the returned stream should only contain non-empty Optionals holding non-null Strings.

Why is findFirst() throwing a NullPointerException then?

Optional<String> cookie = 
  Stream.of(headers.get(HttpHeaders.SET_COOKIE), headers.get(HttpHeaders.COOKIE))
                        .flatMap(Collection::stream)
                        .filter(s -> s.contains("identifier"))
                        .map(this::parseCookieValue) //returns an Optional<String> from Optional.ofNullable(), null-values should result in empty Optionals
                        .filter(Optional::isPresent) // filters out non-present values
                        .map(Optional::get) // all Optionals here should have values
                        .findFirst(); // so why is this still throwing a NullPointerException?

Stacktrace:

Caused by: java.lang.NullPointerException
    at com.example.services.impl.RestServiceImpl$$Lambda$11/873175411.apply(Unknown Source)
    at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267)
    at java.util.Spliterators$ArraySpliterator.tryAdvance(Spliterators.java:958)
    at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
    at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:529)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:516)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:502)
    at java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:464)
    at com.example.services.impl.RestServiceImpl.login(RestServiceImpl.java:81)

Line 81 is the findFirst()-method call.

Answer 1

Reading exceptions which appear inside the Stream API is not trivial. First thing you should not forget is that Stream is lazy: everything is actually executed inside the terminal operation. Thus in your case the whole Stream processing is executed inside the findFirst call and if you see the NullPointerException it can be produced by any step of your pipeline, not just by findFirst itself. Let's take a closer look to the top of stacktrace:

Caused by: java.lang.NullPointerException
    at com.example.services.impl.RestServiceImpl$$Lambda$11/873175411.apply(Unknown Source)
    at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267)
    at java.util.Spliterators$ArraySpliterator.tryAdvance(Spliterators.java:958)
    at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)

If you have some Spliterator.tryAdvance or Spliterator.forEachRemaining call in the trace, then the exception actually occurred during the processing of some stream element, not during the final operations. Here's how the exception looks like if you actually pass the null value to the findFirst:

Exception in thread "main" java.lang.NullPointerException
    at java.util.Objects.requireNonNull(Objects.java:203)
    at java.util.Optional.<init>(Optional.java:96)
    at java.util.Optional.of(Optional.java:108)
    at java.util.stream.FindOps$FindSink$OfRef.get(FindOps.java:193)
    at java.util.stream.FindOps$FindSink$OfRef.get(FindOps.java:190)
    at java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:464)

See, no spliterator calls here: it finished per-element processing and throws after that.

The topmost stackframe in your case reads as com.example.services.impl.RestServiceImpl$$Lambda$11/873175411.apply. The NullPointerException inside the autogenerated lambda which does not point to any known code usually means that unbound method reference is called for null this argument. To make this more clear you can replace all the method references in your code with lambdas as they actually have a source line:

Optional<String> cookie = 
  Stream.of(headers.get(HttpHeaders.SET_COOKIE), headers.get(HttpHeaders.COOKIE))
                        .flatMap(c -> c.stream())
                        .filter(s -> s.contains("identifier"))
                        .map(c -> this.parseCookieValue(c))
                        .filter(opt -> opt.isPresent())
                        .map(opt -> opt.get())
                        .findFirst();

Now you will see the additional frame with the line number:

Exception in thread "main" java.lang.NullPointerException
    at com.example.services.impl.RestServiceImpl.lambda$0(RestServiceImpl.java:14)
    at com.example.services.impl.RestServiceImpl$$Lambda$1/2055281021.apply(Unknown Source)
    at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267)
    at java.util.Spliterators$ArraySpliterator.tryAdvance(Spliterators.java:958)
    at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)

This line number points exactly to the .flatMap(c -> c.stream()) line showing the cause of your exception.

If you don't want to convert all the suspicious method references to lambdas, you may have a clue looking into the previous frame (ReferencePipeline.java:267). This line in JDK source appears inside the flatMap implementation, so you may conclude that something wrong happens on the flatMap step.

So to summarize:

• If you see the exception involving the terminal Stream operation, it may actually occur on any stage of your Stream.
• When per-element processing is performed, it's likely that you will see tryAdvance or forEachRemaining spliterator method call in the trace. When you don't see it, it's likely that per-element processing is already finished or not started.
• Check the topmost frame first: it may point to the body of your lambda where the exception actually occurs.
• If the topmost frame is somewhat cryptic / has "Unknown Source", it's possible, that you try to bind the method reference to the null pointer. In this case replacing method references with lambdas may help to understand what's going on.
• Don't be afraid looking into the Stream API source. It may also provide a clue.

Answer 2

I found the mistake, and the commenters were correct: the issue was not the Optional, but the source lists! HttpHeaders.get(Object key) returns null if the key is not found. I falsely assumed that somehow null-lists were not collected, or that empty lists are returned instead of null. If I filter for that (or check priorly if the headers exist), it works as expected.

Thanks for pointing me to it! I wrote a small example demonstrating the issue for anyone interested:

package com.example;

import java.util.*;
import java.util.stream.Stream;

public class Main {

    public static void main(String[] args) {
        succeeds();
        fixed();
        fails();
    }

    private static void succeeds() {
        List<String> list1 = Collections.singletonList("identifier=xxx");
        List<String> list2 = Collections.emptyList();
        Optional<String> cookieValue =
                Stream.of(list1, list2)
                        .flatMap(Collection::stream)
                        .filter(s -> s.contains("identifier"))
                        .map(Main::parseCookieValue)
                        .filter(Optional::isPresent)
                        .map(Optional::get)
                        .findFirst();
        System.out.println(cookieValue.orElse("Code works as expected with non-null Lists"));
    }

    private static void fails() {
        List<String> list1 = Collections.singletonList("identifier=xxx");
        List<String> list2 = null;
        Optional<String> cookieValue =
                Stream.of(list1, list2)
                        .flatMap(Collection::stream)
                        .filter(s -> s.contains("identifier"))
                        .map(Main::parseCookieValue)
                        .filter(Optional::isPresent)
                        .map(Optional::get)
                        .findFirst();
        System.out.println(cookieValue.orElse("Exception thrown prior to this call!"));
    }

    private static void fixed() {
        List<String> list1 = Collections.singletonList("identifier=xxx");
        List<String> list2 = null;
        Optional<String> cookieValue =
                Stream.of(list1, list2)
                        .filter(l -> l != null)
                        .flatMap(Collection::stream)
                        .filter(s -> s.contains("identifier"))
                        .map(Main::parseCookieValue)
                        .filter(Optional::isPresent)
                        .map(Optional::get)
                        .findFirst();
        System.out.println(cookieValue.orElse("Code works as expected after null Lists have been filtered"));
    }

    private static Optional<String> parseCookieValue(final String headerString) {
        System.out.println("Parsing method called");
        //return an empty Optional for testing;
        return Optional.empty();
    }
}