When parsing JSON with Aeson, why is Maybe treated differently when it's in a type parameter?

Question

Suppose we have some data classes

{-# LANGUAGE DeriveGeneric, DuplicateRecordFields #-}

import Data.Aeson
import Data.ByteString.Lazy.Char8
import GHC.Generics

data Foo a = Foo { payload :: a }
    deriving (Show, Generic)

instance ToJSON a => ToJSON (Foo a)
instance FromJSON a => FromJSON (Foo a)

data Bar a = Bar { payload :: Maybe a }
    deriving (Show, Generic)

instance ToJSON a => ToJSON (Bar a)
instance FromJSON a => FromJSON (Bar a)

Then we try to decode as following:

*Main > decode $ pack "{}" :: Maybe (Bar String)
Just (Foo {payload = Nothing})
*Main > decode $ pack "{}" :: Maybe (Foo (Maybe String))
Nothing

So why can't we decode JSON in the last try? The data classes seem to be the same, and they both work the same way with toJSON:

*Main > toJSON $ Foo (Nothing :: Maybe String)
Object (fromList [("payload",Null)])
*Main > toJSON $ Bar (Nothing :: Maybe String)
Object (fromList [("payload",Null)])

Answer 1

Updated: with a simple solution at the bottom.

This is confusing, but it's working more or less as designed. You could try submitting it as an aeson issue, but I suspect it will be closed as "won't fix".

What's happening is that the generic instance generated for FromJSON (Bar a) is equivalent to:

instance FromJSON a => FromJSON (Bar a) where
  parseJSON = withObject "Bar" $ \v -> Bar
    <$> v .:? "payload"

Note the use of the (.:?) operator generated because of the Maybe a field in Bar. In a structure with a mixture of Maybe and non-Maybe fields, there would be a corresponding mixture of (.:?) and (.:) operators.

Note that this instance is generated once and for all for every possible a. The reason it's polymorphic is that the (.:?) implementation can dispatch to the parseJSON method in the FromJSON a dictionary supplied by the instance constraint. Also note that the only reason we can use (.:?) is that it's known at compile time that for all possible types a, the field payload in the Bar object has type Maybe a and so use of the (.:?) operator will typecheck.

Now, consider the instance generated for FromJSON (Foo a). This will be equivalent to:

instance FromJSON a => FromJSON (Foo a) where
  parseJSON = withObject "Foo" $ \v -> Foo
    <$> v .: "payload"

It's exactly analogous to the Bar a instance above, except it uses the (.:) operator. Again, it has a single implementation at compile time that works for every possible a by dispatching to parseJSON in the FromJSON a dictionary. There's no way that this instance can use the (.:?) operator, since general a and Maybe t can't unify, and there's no way that it can somehow "inspect" the type a, whether at compile time or runtime, to see if it's a Maybe, for much the same reason that you can't write a total polymorphic function with type a -> a that's anything other than the identity.

Therefore, this Foo a instance can't make the payload field optional! Instead, it has to treat payload as mandatory and -- when used to parse a Foo (Maybe String) -- dispatch to a FromJSON t => FromJSON (Maybe t) instance (which allows null but otherwise dispatches to the FromJSON String instance).

Now, why does it seemingly work fine for ToJSON? Well, the instances for both ToJSON (Foo a) and ToJSON (Bar a) generate the same sort of (monomorphic) Value representation:

> toJSON (Foo (Nothing :: Maybe String))
Object (fromList [("payload",Null)])
> toJSON (Bar (Nothing :: Maybe String))
Object (fromList [("payload",Null)])

and the removal of null fields takes place uniformly when this value is encoded to JSON.

This leads to an unfortunate asymmetry in the FromJSON and ToJSON instances, but that's what's going on.

And I just realized I forgot to share the simple solution for fixing it. Just define two generic instances for Foo, one overlapping instance to handle Maybes and the other for other types:

instance {-# OVERLAPPING #-} FromJSON a => FromJSON (Foo (Maybe a))
instance FromJSON a => FromJSON (Foo a)