Strange NPE with io.circe.Decoder

Question

I have 2 variables declared as follows,

 implicit val decodeURL: Decoder[URL] = Decoder.decodeString.emapTry(s => Try(new URL(s)))     // #1

 implicit val decodeCompleted = Decoder[List[URL]].prepare(_.downField("completed")) // #2

Both lines compile and run.

However, if I annotate #2 with type i.e. implicit val decodeCompleted: Decoder[List[URL]] = Decoder[List[URL]].prepare(_.downField("completed")). It compiles and #2 will throw NullPointerException (NPE) during runtime.

How could this happen? I don't know if this is Circe or just plain Scala issue. Why #2 is different from #1? Thanks

Answer 1

The issue is that you are supposed to use implicits with annotation always.

Now, when you use them when they are not annotated you get into some sort on undefined/invalid behavior zone. That is why with unannotated happens sth like this:

• I want to use Decoder[List[URL]]
• there is no (annotated) Decoder[List[URL]] implicit in the scope
• let's derive it normally (no need for generic.auto._ because the definition for that is in a companion object)
• once derived you call on it .prepare(_.downField("completed"))
• the final result is of type Decoder[List[URL]], so that is inferred type of decodeCompleted

Now, what happens if you annotate?

• I want to use Decoder[List[URL]]
• there is decodeCompleted declared as something that fulfills that definition
• let use decodeCompleted value
• but decodeCompleted wasn't initialized! in fact we are initializing it right now!
• as a result you end up with decodeCompleted = null

This is virtually equal to:

val decodeCompleted = decodeCompleted

except that the layer of indirection get's in the way of discovering the absurdity of this by compiler. (If you replaced val with def you would end up with an infinite recursion and stack overflow):

@ implicit val s: String = implicitly[String] 
s: String = null

@ implicit def s: String = implicitly[String] 
defined function s

@ s 
java.lang.StackOverflowError
  ammonite.$sess.cmd1$.s(cmd1.sc:1)
  ammonite.$sess.cmd1$.s(cmd1.sc:1)
  ammonite.$sess.cmd1$.s(cmd1.sc:1)
  ammonite.$sess.cmd1$.s(cmd1.sc:1)

Yup, its messed up by compiler. You did nothing wrong and in a perfect world it would work.

Scala community mitigates that by distincting:

• auto derivation - when you need implicit somewhere and it is automatically derived without you defining a variable for it
• semiauto derivation - when you derive into a value, and make that value implicit

In the later case, you usually have some utilities like:

import io.circe.generic.semiauto._

implicit val decodeCompleted: Decoder[List[URL]] = deriveDecoder[List[URL]]

It works because it takes DerivedDecoder[A] implicit and then it extracts Decoder[A] from it, so you never end up with implicit val a: A = implicitly[A] scenario.

Answer 2

Indeed the problem is that you introduce a recursive val, like @MateuszKubuszok explained.

The most straightforward—although slightly ugly—workaround is:

implicit val decodeCompleted: Decoder[List[URL]] = {
  val decodeCompleted = null
  Decoder[List[URL]].prepare(_.downField("completed"))
}

By shadowing decodeCompleted in the right-hand side, implicit search will no longer consider it as a candidate inside that code block, because it can no longer be referenced.