FSharp.Data -- How to Handle Null JSON Values?

Question

I'm trying to convert the FSharp.Data examples to solutions for my problem i'm dealing with, but i'm just not getting very far.

Problem

Given an endpoint that returns json similar to:

{
  Products:[{
    Id:43,
    Name:"hi"
  },
  {
    Id:45,
    Name:"other prod"
  }
  ]
}

How can i load the data and then only get the Ids out of real, existing data?

I dont understand how to "pattern match out" the possibilities that:

• it could return nothing
• that root.Products could be not existing/empty
• that Id might not exist

Attempt Via Null Matching

namespace Printio

open System 
open FSharp.Data
open FSharp.Data.JsonExtensions

module PrintioApi =
    type ApiProducts = JsonProvider<"https://api.print.io/api/v/1/source/widget/products?recipeId=f255af6f-9614-4fe2-aa8b-1b77b936d9d6&countryCode=US">

let getProductIds url =
    async {
        let! json = ApiProducts.AsyncLoad url
        let ids = match json with
            | null  -> [||]
            | _ -> 
                match json.Products with
                    | null -> [||]
                    | _ -> Array.map (fun (x:ApiProducts.Product)-> x.Id) json.Products

        return ids
        }

Answer 1

If you suspect that your data source may contain some missing values, you can set SampleIsList = true in the JsonProvider, and give it a list of samples, instead of a single example:

open FSharp.Data

type ApiProducts = JsonProvider<"""
[
    {
        "Products": [{
            "Id": 43,
            "Name": "hi"
        }, {
            "Name": "other prod"
        }]
    },
    {}
]
""", SampleIsList = true>

As Gustavo Guerra also hints in his answer, Products is already a list, so you can supply one example of a product that has an Id (the first one), and one example that doesn't have an Id (the second one).

Likewise, you can give an example where Products is entirely missing. Since the root object contains no other data, this is simply the empty object: {}.

The JsonProvider is intelligent enough to interpret a missing Products property as an empty array.

Since a product may or may not have an Id, this property is inferred to have the type int option.

You can now write a function that takes a JSON string as input and gives you all the IDs it can find:

let getProductIds json =
    let root = ApiProducts.Parse json
    root.Products |> Array.choose (fun p -> p.Id)

Notice that it uses Array.choose instead of Array.map, since Array.choose automatically chooses only those Id values that are Some.

You can now test with various values to see that it works:

> getProductIds """{ "Products": [{ "Id": 43, "Name": "hi" }, { "Id": 45, "Name": "other prod"  }] }""";;
> val it : int [] = [|43; 45|]

> getProductIds """{ "Products": [{ "Id": 43, "Name": "hi" }, { "Name": "other prod" }] }""";;
> val it : int [] = [|43|]

> getProductIds "{}";;
> val it : int [] = [||]

It still crashes on empty input, though; if there's a TryParse function or similar for JsonProvider, I haven't found it yet...

Answer 2

You probably don't need pattern matching for checking whether it's an empty array of not if you have some level of confidence in the source data. Something like this might just work fine: -

let getProductIds url =
    async {
        let! json = ApiProducts.AsyncLoad url
        return json.Products |> Seq.map(fun p -> p.Id) |> Seq.cache
    }

Note you shouldn't use Async.RunSynchronously when in an async { } block - you can do a let! binding which will await the result asynchronously.

Answer 3

Give the type provider enough examples to infer those cases. Example:

[<Literal>]
let sample = """
{
  Products:[{
    Id:null,
    Name:"hi"
  },
  {
    Id:45,
    Name:"other prod"
  }
  ]
}
"""

type MyJsonType = JsonProvider<sample>

But do note it will never be 100% safe if the json is not regular enough