Read a JSON file and generate string keys with values in a dictionary like object

Question

I have to read in a JSON file that will be fairly big, with potentially hundreds of key value pairs and nested JSON key/values.

So say I have something like this:

 {
      "abc": {
        "123": {
          "donkey": "hello world",  
          "kong": 123    
        },
        "meta": {
          "aaa": "bbb"      
        }
      }
    }

I want to read this JSON file, and then initialize a Dictionary with keys that would be like this (based on the above JSON file):

"abc.123.donkey": "hello world"
"abc.123.kong": 123
"abc.meta.aaa": "bbb"

So basically the key is like a namespace based on the number of nested items, and then it has the value.

How should I go about parsing the JSON file when I don't know the shape of the JSON ahead of time, given that I need to create a dictionary out of it using this namespace style keys?

Answer 1

A. System.Text.Json

I'm sure this will get easier, but as of .NET Core 3.0 JsonDocument is a way.

using System.Linq;
using System.Text.Json;
(...)


public static Dictionary<string, JsonElement> GetFlat(string json)
{
    IEnumerable<(string Path, JsonProperty P)> GetLeaves(string path, JsonProperty p)
        => p.Value.ValueKind != JsonValueKind.Object
            ? new[] { (Path: path == null ? p.Name : path + "." + p.Name, p) }
            : p.Value.EnumerateObject() .SelectMany(child => GetLeaves(path == null ? p.Name : path + "." + p.Name, child));

    using (JsonDocument document = JsonDocument.Parse(json)) // Optional JsonDocumentOptions options
        return document.RootElement.EnumerateObject()
            .SelectMany(p => GetLeaves(null, p))
            .ToDictionary(k => k.Path, v => v.P.Value.Clone()); //Clone so that we can use the values outside of using
}

More expressive version is shown below.

Test

using System.Linq;
using System.Text.Json;
(...)

var json = @"{
    ""abc"": {
        ""123"": {
            ""donkey"": ""hello world"",
            ""kong"": 123
        },
        ""meta"": {
            ""aaa"": ""bbb""
        }
    }
}";

var d = GetFlat(json);
var options2 = new JsonSerializerOptions { WriteIndented = true };
Console.WriteLine(JsonSerializer.Serialize(d, options2));

Output

{
  "abc.123.donkey": "hello world",
  "abc.123.kong": 123,
  "abc.meta.aaa": "bbb"
}

More expressive version

using System.Linq;
using System.Text.Json;
(...)

static Dictionary<string, JsonElement> GetFlat(string json)
{
    using (JsonDocument document = JsonDocument.Parse(json))
    {
        return document.RootElement.EnumerateObject()
            .SelectMany(p => GetLeaves(null, p))
            .ToDictionary(k => k.Path, v => v.P.Value.Clone()); //Clone so that we can use the values outside of using
    }
}


static IEnumerable<(string Path, JsonProperty P)> GetLeaves(string path, JsonProperty p)
{
    path = (path == null) ? p.Name : path + "." + p.Name;
    if (p.Value.ValueKind != JsonValueKind.Object)
        yield return (Path: path, P: p);
    else
        foreach (JsonProperty child in p.Value.EnumerateObject())
            foreach (var leaf in GetLeaves(path, child))
                yield return leaf;
}

B. Json.NET

You can do with Json.NET.

---

B1. SelectTokens

using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
(...)

JObject jo = JObject.Parse(json);
var d = jo.SelectTokens("$..*")
    .Where(t => t.HasValues == false)
    .ToDictionary(k => k.Path, v => v);

Console.WriteLine(JsonConvert.SerializeObject(d, Formatting.Indented));

Output

{
  "abc.123.donkey": "hello world",
  "abc.123.kong": 123,
  "abc.meta.aaa": "bbb"
}

B2. JToken traversal

using Newtonsoft.Json;
using Newtonsoft.Json.Linq;

(...)

static IEnumerable<JToken> Traverse(JToken jo)
{
    if (!jo.Any()) yield return jo;
    foreach (var ch in jo)
        foreach (var x in Traverse(ch))
            yield return x;
}

Test

var json = @"{
""abc"": {
    ""123"": {
        ""donkey"": ""hello world"",
        ""kong"": 123
    },
    ""meta"": {
        ""aaa"": ""bbb""
    }
}
}";

JObject jo = JObject.Parse(json);
var d = Traverse(jo).ToDictionary(k => k.Path, v => v);
var json2 = JsonConvert.SerializeObject(d, Formatting.Indented);
Console.WriteLine(json2);

Output

{
  "abc.123.donkey": "hello world",
  "abc.123.kong": 123,
  "abc.meta.aaa": "bbb"
}

Answer 2

You can use Json.NET's LINQ to JSON feature to accomplish this.

You can deserialize the object to a JObject, which is a dictionary with better typings for json. A JToken is the base type for any json value.

Once you have the JObject you can iterate its values. If it's another JObject then navigate it recursively, otherwise save the current value to the result dictionary.

After the whole tree is visited we return the flattened result.

Executing the following program:

using System;
using System.Collections.Generic;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;

namespace FlattenObject
{
    class Program
    {
        static void Main(string[] args)
        {
            var json = @"
{
  ""abc"": {
    ""123"": {
      ""donkey"": ""hello world"",
      ""kong"": 123
    },
    ""meta"": {
      ""aaa"": ""bbb""
    }
  }
}
";
            var root = JsonConvert.DeserializeObject<JObject>(json);
            var flattened = Flatten(root);
            Console.WriteLine(JsonConvert.SerializeObject(flattened, Formatting.Indented));
        }

        static Dictionary<string, JToken> Flatten(JObject root)
        {
            var result = new Dictionary<string, JToken>();
            void FlattenRec(string path, JToken value)
            {
                if (value is JObject dict)
                {
                    foreach (var pair in dict)
                    {
                        string joinedPath = path != null
                            ? path + "." + pair.Key
                            : pair.Key;
                        FlattenRec(joinedPath, pair.Value);
                    }
                }
                else
                {
                    result[path] = value;
                }
            }

            FlattenRec(null, root);
            return result;
        }
    }
}

gives output:

{
  "abc.123.donkey": "hello world",
  "abc.123.kong": 123,
  "abc.meta.aaa": "bbb"
}

Note: the code above uses local functions which is a recent feature. If you can't use it then create a helper method and pass the result dictionary explicitly.