Customizing Json.NET serialization: turning object into array to avoid repetition of property names

Question

I'm sending large amounts of different JSON graphs from a server to a client (I control both) and they all contain a pathological case: a large array of homogeneous (same type) values. So, for example, part of the payload looks like:

[{"LongPropertyName":87, "AnotherVeryLongPropertyName":93,   "BlahBlahBlahBlahBlah": 78},  {"LongPropertyName":97, "AnotherVeryLongPropertyName":43,   "BlahBlahBlahBlahBlah": 578},  {"LongPropertyName":92, "AnotherVeryLongPropertyName":-3,   "BlahBlahBlahBlahBlah": 817}, ... 

I've added some formatting, but as you can see, it's ridiculous from a Huffman coding point of view, ie that common things should be efficiently expressed.

So, since I control both the deserialization and the serialization ends, I want to implement a transform where this:

[{"Key1":87,"Key2":99},{"Key1":42,"Key2":-8}] 

gets turned into something like this:

[["$","Key1","Key2"],[87,99],[42,-8]] 

which as you can see is more compact even with just two objects.

Where do I hook into Json.NET to do this transformation? I want to do this automatically for as many objects as possible. I've found ContractResolvers but I'm not sure if they're happening at the stage I want - I'm not sure how to use its methods to turn a JSON object/dictionary into an array.

Alternatively, if a similar thing has already been implemented for Json.NET, I'd want to use that instead. But I am not confused about the sort of change I want to make (see above), just where I'd hook into Json.NET to make it happen.

(I have tried gzipping it. It works fine and shaves off between 70% and 95%, but it still has to output the full JSON text and do all that compression/decompression. This question is: how do I just output a more compact form of the data from the beginning?)

---

Update: The way you do this is with a JsonConverter. I had already written several but for some reason I thought they would conflict.

What I ended up with was Brian Rogers' base along with some changes to also embed/flatten any directly contained objects. This was not part of the original question, but the reason I did that is because if I had:

[{"A": 42,"B":{"PropOne":87,"PropTwo":93,"PropThree":78}}, {"A":-72,"B":{"PropOne":97,"PropTwo":43,"PropThree":578}] 

...I ended up with:

[["A","B"],[42,{"PropOne":87,"PropTwo":93,"PropThree":78}], [-72,{"PropOne":97,"PropTwo":43,"PropThree":578}]] 

...and that doesn't really save anything. Whereas if I embedded/flattened the object as its constituent keys, I end up with:

[["A","B_PropOne","B_PropTwo","B_PropThree"],[42,87,93,78],[-72,97,43,578]] 

Answer 1

I believe the best way to achieve what you are looking for is to use a custom JsonConverter as was suggested by @Ilija Dimov. His converter is a good start, and should work fine for certain cases, but you may run into trouble if you are serializing a more complex graph of objects. I offer the following converter as an alternative solution. This converter has the following advantages:

• Uses the Json.Net's built-in serialization logic for the list items, so that any attributes applied to the classes are respected, including [JsonConstructor] and [JsonProperty]. Other converters are respected as well.
• Ignores lists of primitives and strings so that these are serialized normally.
• Supports List<YourClass> where YourClass contains complex objects, including List<YourOtherClass>.

Limitations:

• Does not currently support lists of anything enumerable, e.g. List<List<YourClass>> or List<Dictionary<K, YourClass>>, but could be modified to do so if needed. These will be serialized in the usual way for now.

Here is the code for the converter:

class ListCompactionConverter : JsonConverter {     public override bool CanConvert(Type objectType)     {         // We only want to convert lists of non-enumerable class types (including string)         if (objectType.IsGenericType && objectType.GetGenericTypeDefinition() == typeof(List<>))         {             Type itemType = objectType.GetGenericArguments().Single();             if (itemType.IsClass && !typeof(IEnumerable).IsAssignableFrom(itemType))             {                 return true;             }         }         return false;     }      public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)     {         JArray array = new JArray();         IList list = (IList)value;         if (list.Count > 0)         {             JArray keys = new JArray();              JObject first = JObject.FromObject(list[0], serializer);             foreach (JProperty prop in first.Properties())             {                 keys.Add(new JValue(prop.Name));             }             array.Add(keys);              foreach (object item in list)             {                 JObject obj = JObject.FromObject(item, serializer);                 JArray itemValues = new JArray();                 foreach (JProperty prop in obj.Properties())                 {                     itemValues.Add(prop.Value);                 }                 array.Add(itemValues);             }         }         array.WriteTo(writer);     }      public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)     {         IList list = (IList)Activator.CreateInstance(objectType);  // List<T>         JArray array = JArray.Load(reader);         if (array.Count > 0)         {             Type itemType = objectType.GetGenericArguments().Single();              JArray keys = (JArray)array[0];             foreach (JArray itemValues in array.Children<JArray>().Skip(1))             {                 JObject item = new JObject();                 for (int i = 0; i < keys.Count; i++)                 {                     item.Add(new JProperty(keys[i].ToString(), itemValues[i]));                 }                  list.Add(item.ToObject(itemType, serializer));             }         }         return list;     } } 

Below is a full round-trip demo using this converter. We have a list of mutable Company objects which each contain a list of immutable Employees. For demonstration purposes, each company also has a simple list of string aliases using a custom JSON property name, and we also use an IsoDateTimeConverter to customize the date format for the employee HireDate. The converters are passed to the serializer via the JsonSerializerSettings class.

class Program {     static void Main(string[] args)     {         List<Company> companies = new List<Company>         {             new Company             {                 Name = "Initrode Global",                 Aliases = new List<string> { "Initech" },                 Employees = new List<Employee>                 {                     new Employee(22, "Bill Lumbergh", new DateTime(2005, 3, 25)),                     new Employee(87, "Peter Gibbons", new DateTime(2011, 6, 3)),                     new Employee(91, "Michael Bolton", new DateTime(2012, 10, 18)),                 }             },             new Company             {                 Name = "Contoso Corporation",                 Aliases = new List<string> { "Contoso Bank", "Contoso Pharmaceuticals" },                 Employees = new List<Employee>                 {                     new Employee(23, "John Doe", new DateTime(2007, 8, 22)),                     new Employee(61, "Joe Schmoe", new DateTime(2009, 9, 12)),                 }             }         };          JsonSerializerSettings settings = new JsonSerializerSettings();         settings.Converters.Add(new ListCompactionConverter());         settings.Converters.Add(new IsoDateTimeConverter { DateTimeFormat = "dd-MMM-yyyy" });         settings.Formatting = Formatting.Indented;          string json = JsonConvert.SerializeObject(companies, settings);         Console.WriteLine(json);         Console.WriteLine();          List<Company> list = JsonConvert.DeserializeObject<List<Company>>(json, settings);         foreach (Company c in list)         {             Console.WriteLine("Company: " + c.Name);             Console.WriteLine("Aliases: " + string.Join(", ", c.Aliases));             Console.WriteLine("Employees: ");             foreach (Employee emp in c.Employees)             {                 Console.WriteLine("  Id: " + emp.Id);                 Console.WriteLine("  Name: " + emp.Name);                 Console.WriteLine("  HireDate: " + emp.HireDate.ToShortDateString());                 Console.WriteLine();             }             Console.WriteLine();         }     } }  class Company {     public string Name { get; set; }     [JsonProperty("Doing Business As")]     public List<string> Aliases { get; set; }     public List<Employee> Employees { get; set; } }  class Employee {     [JsonConstructor]     public Employee(int id, string name, DateTime hireDate)     {         Id = id;         Name = name;         HireDate = hireDate;     }     public int Id { get; private set; }     public string Name { get; private set; }     public DateTime HireDate { get; private set; } } 

Here is the output from the above demo, showing the intermediate JSON as well as the contents of the objects deserialized from it.

[   [     "Name",     "Doing Business As",     "Employees"   ],   [     "Initrode Global",     [       "Initech"     ],     [       [         "Id",         "Name",         "HireDate"       ],       [         22,         "Bill Lumbergh",         "25-Mar-2005"       ],       [         87,         "Peter Gibbons",         "03-Jun-2011"       ],       [         91,         "Michael Bolton",         "18-Oct-2012"       ]     ]   ],   [     "Contoso Corporation",     [       "Contoso Bank",       "Contoso Pharmaceuticals"     ],     [       [         "Id",         "Name",         "HireDate"       ],       [         23,         "John Doe",         "22-Aug-2007"       ],       [         61,         "Joe Schmoe",         "12-Sep-2009"       ]     ]   ] ]  Company: Initrode Global Aliases: Initech Employees:   Id: 22   Name: Bill Lumbergh   HireDate: 3/25/2005    Id: 87   Name: Peter Gibbons   HireDate: 6/3/2011    Id: 91   Name: Michael Bolton   HireDate: 10/18/2012   Company: Contoso Corporation Aliases: Contoso Bank, Contoso Pharmaceuticals Employees:   Id: 23   Name: John Doe   HireDate: 8/22/2007    Id: 61   Name: Joe Schmoe   HireDate: 9/12/2009 

I've added a fiddle here in case you'd like to play with the code.