Custom JSON serializer for optional property with System.Text.Json

Question

I'm trying to implement a JSON serialization mechanism which handles both null and missing JSON values, to be able to perform partial updates when needed (so that it does not touch the field in the database when the value is missing, but it clears it when the value is explicitly set to null).

I created a custom struct copied from Roslyn's Optional<T> type:

public readonly struct Optional<T>
{
    public Optional(T value)
    {
        this.HasValue = true;
        this.Value = value;
    }

    public bool HasValue { get; }
    public T Value { get; }
    public static implicit operator Optional<T>(T value) => new Optional<T>(value);
    public override string ToString() => this.HasValue ? (this.Value?.ToString() ?? "null") : "unspecified";
}

Now I want to be able to serialize/deserialize to/from JSON so that any missing field in JSON is preserved when roundtripping it through the Optional<T> object:

public class CustomType
{
    [JsonPropertyName("foo")]
    public Optional<int?> Foo { get; set; }

    [JsonPropertyName("bar")]
    public Optional<int?> Bar { get; set; }

    [JsonPropertyName("baz")]
    public Optional<int?> Baz { get; set; }
}

Then:

var options = new JsonSerializerOptions();
options.Converters.Add(new OptionalConverter());

string json = @"{""foo"":0,""bar"":null}";
CustomType parsed = JsonSerializer.Deserialize<CustomType>(json, options);
string roundtrippedJson = JsonSerializer.Serialize(parsed, options);

// json and roundtrippedJson should be equivalent
Console.WriteLine("json:             " + json);
Console.WriteLine("roundtrippedJson: " + roundtrippedJson);

I started an implementation based on JsonConverterFactory, but I can't seem to find a proper way to omit the property during serialization if the optional's HasValue is false:

public class OptionalConverter : JsonConverterFactory
{
    public override bool CanConvert(Type typeToConvert)
    {
        if (!typeToConvert.IsGenericType) { return false; }
        if (typeToConvert.GetGenericTypeDefinition() != typeof(Optional<>)) { return false; }
        return true;
    }

    public override JsonConverter CreateConverter(Type typeToConvert, JsonSerializerOptions options)
    {
        Type valueType = typeToConvert.GetGenericArguments()[0];

        return (JsonConverter)Activator.CreateInstance(
            type: typeof(OptionalConverterInner<>).MakeGenericType(new Type[] { valueType }),
            bindingAttr: BindingFlags.Instance | BindingFlags.Public,
            binder: null,
            args: null,
            culture: null
        );
    }

    private class OptionalConverterInner<T> : JsonConverter<Optional<T>>
    {
        public override Optional<T> Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
        {
            T value = JsonSerializer.Deserialize<T>(ref reader, options);
            return new Optional<T>(value);
        }

        public override void Write(Utf8JsonWriter writer, Optional<T> value, JsonSerializerOptions options)
        {
            // Does not work (produces invalid JSON).
            // Problem: the object's key has already been written in the JSON writer at this point.
            if (value.HasValue)
            {
                JsonSerializer.Serialize(writer, value.Value, options);
            }
        }
    }
}

Problem: this produces the following output, which is invalid:

json:             {"foo":0,"bar":null}
roundtrippedJson: {"foo":0,"bar":null,"baz":}

How can I solve this?

Answer 1

A custom JsonConverter<T> cannot prevent the serialization of a value to which the converter applies, see [System.Text.Json] Converter-level conditional serialization #36275 for confirmation.

In .Net 5 there is an option to ignore default values which should do what you need, see How to ignore properties with System.Text.Json. This version introduces JsonIgnoreCondition.WhenWritingDefault:

public enum JsonIgnoreCondition
{
    // Property is never ignored during serialization or deserialization.
    Never = 0,
    // Property is always ignored during serialization and deserialization.
    Always = 1,
    // If the value is the default, the property is ignored during serialization.
    // This is applied to both reference and value-type properties and fields.
    WhenWritingDefault = 2,
    // If the value is null, the property is ignored during serialization.
    // This is applied only to reference-type properties and fields.
    WhenWritingNull = 3,
}

You will be able to apply the condition to specific properties via [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingDefault)] or globally by setting JsonSerializerOptions.DefaultIgnoreCondition.

Thus in .Net 5 your class would look like:

public class CustomType
{
    [JsonPropertyName("foo")]
    [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingDefault)]
    public Optional<int?> Foo { get; set; }

    [JsonPropertyName("bar")]
    [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingDefault)]
    public Optional<int?> Bar { get; set; }

    [JsonPropertyName("baz")]
    [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingDefault)]
    public Optional<int?> Baz { get; set; }
}

And the HasValue check should be removed from OptionalConverterInner<T>.Write():

public override void Write(Utf8JsonWriter writer, Optional<T> value, JsonSerializerOptions options) =>
    JsonSerializer.Serialize(writer, value.Value, options);

Demo fiddle #1 here.

In .Net 3, as there is no conditional serialization mechanism in System.Text.Json, your only option to conditionally omit optional properties without a value is to write a custom JsonConverter<T> for all classes that contain optional properties. This is not made easy by the fact that JsonSerializer does not provide any access to its internal contract information so we need to either handcraft a converter for each and every such type, or write our own generic code via reflection.

Here is one attempt to create such generic code:

public interface IHasValue
{
    bool HasValue { get; }
    object GetValue();
}

public readonly struct Optional<T> : IHasValue
{
    public Optional(T value)
    {
        this.HasValue = true;
        this.Value = value;
    }

    public bool HasValue { get; }
    public T Value { get; }
    public object GetValue() => Value;
    public static implicit operator Optional<T>(T value) => new Optional<T>(value);
    public override string ToString() => this.HasValue ? (this.Value?.ToString() ?? "null") : "unspecified";
}

public class TypeWithOptionalsConverter<T> : JsonConverter<T> where T : class, new()
{
    class TypeWithOptionalsConverterContractFactory : JsonObjectContractFactory<T>
    {
        protected override Expression CreateSetterCastExpression(Expression e, Type t)
        {
            // (Optional<Nullable<T>>)(object)default(T) does not work, even though (Optional<Nullable<T>>)default(T) does work.
            // To avoid the problem we need to first cast to Nullable<T>, then to Optional<Nullable<T>>
            if (t.IsGenericType && t.GetGenericTypeDefinition() == typeof(Optional<>))
                return Expression.Convert(Expression.Convert(e, t.GetGenericArguments()[0]), t);
            return base.CreateSetterCastExpression(e, t);
        }
    }
    
    static readonly TypeWithOptionalsConverterContractFactory contractFactory = new TypeWithOptionalsConverterContractFactory();
    
    public override T Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
    {
        var properties = contractFactory.GetProperties(typeToConvert);

        if (reader.TokenType == JsonTokenType.Null)
            return null;
        if (reader.TokenType != JsonTokenType.StartObject)
            throw new JsonException();
        var value = new T();
        while (reader.Read())
        {
            if (reader.TokenType == JsonTokenType.EndObject)
                return value;
            if (reader.TokenType != JsonTokenType.PropertyName)
                throw new JsonException();
            string propertyName = reader.GetString();
            if (!properties.TryGetValue(propertyName, out var property) || property.SetValue == null)
            {
                reader.Skip();
            }
            else
            {
                var type = property.PropertyType.IsGenericType && property.PropertyType.GetGenericTypeDefinition() == typeof(Optional<>) 
                    ? property.PropertyType.GetGenericArguments()[0] : property.PropertyType;
                var item = JsonSerializer.Deserialize(ref reader, type, options);
                property.SetValue(value, item);
            }
        }
        throw new JsonException();
    }           

    public override void Write(Utf8JsonWriter writer, T value, JsonSerializerOptions options)
    {
        writer.WriteStartObject();
        foreach (var property in contractFactory.GetProperties(value.GetType()))
        {
            if (options.IgnoreReadOnlyProperties && property.Value.SetValue == null)
                continue;
            var item = property.Value.GetValue(value);
            if (item is IHasValue hasValue)
            {
                if (!hasValue.HasValue)
                    continue;
                writer.WritePropertyName(property.Key);
                JsonSerializer.Serialize(writer, hasValue.GetValue(), options);
            }
            else
            {
                if (options.IgnoreNullValues && item == null)
                    continue;
                writer.WritePropertyName(property.Key);
                JsonSerializer.Serialize(writer, item, property.Value.PropertyType, options);
            }
        }
        writer.WriteEndObject();
    }
}

public class JsonPropertyContract<TBase>
{
    internal JsonPropertyContract(PropertyInfo property, Func<Expression, Type, Expression> setterCastExpression)
    {
        this.GetValue = ExpressionExtensions.GetPropertyFunc<TBase>(property).Compile();
        if (property.GetSetMethod() != null)
            this.SetValue = ExpressionExtensions.SetPropertyFunc<TBase>(property, setterCastExpression).Compile();
        this.PropertyType = property.PropertyType;
    }
    public Func<TBase, object> GetValue { get; }
    public Action<TBase, object> SetValue { get; }
    public Type PropertyType { get; }
}

public class JsonObjectContractFactory<TBase>
{
    protected virtual Expression CreateSetterCastExpression(Expression e, Type t) => Expression.Convert(e, t);
    
    ConcurrentDictionary<Type, ReadOnlyDictionary<string, JsonPropertyContract<TBase>>> Properties { get; } = 
        new ConcurrentDictionary<Type, ReadOnlyDictionary<string, JsonPropertyContract<TBase>>>();

    ReadOnlyDictionary<string, JsonPropertyContract<TBase>> CreateProperties(Type type)
    {
        if (!typeof(TBase).IsAssignableFrom(type))
            throw new ArgumentException();
        var dictionary = type
            .GetProperties(BindingFlags.Public | BindingFlags.Instance | BindingFlags.FlattenHierarchy)
            .Where(p => p.GetIndexParameters().Length == 0 && p.GetGetMethod() != null
                   && !Attribute.IsDefined(p, typeof(System.Text.Json.Serialization.JsonIgnoreAttribute)))
            .ToDictionary(p => p.GetCustomAttribute<System.Text.Json.Serialization.JsonPropertyNameAttribute>()?.Name ?? p.Name,
                          p => new JsonPropertyContract<TBase>(p, (e, t) => CreateSetterCastExpression(e, t)), 
                          StringComparer.OrdinalIgnoreCase);
        return dictionary.ToReadOnly();
    }

    public IReadOnlyDictionary<string, JsonPropertyContract<TBase>> GetProperties(Type type) => Properties.GetOrAdd(type, t => CreateProperties(t));
}

public static class DictionaryExtensions
{
    public static ReadOnlyDictionary<TKey, TValue> ToReadOnly<TKey, TValue>(this IDictionary<TKey, TValue> dictionary) => 
        new ReadOnlyDictionary<TKey, TValue>(dictionary ?? throw new ArgumentNullException());
}

public static class ExpressionExtensions
{
    public static Expression<Func<T, object>> GetPropertyFunc<T>(PropertyInfo property)
    {
        // (x) => (object)x.Property;
        var arg = Expression.Parameter(typeof(T), "x");
        var getter = Expression.Property(arg, property);
        var cast = Expression.Convert(getter, typeof(object));
        return Expression.Lambda<Func<T, object>>(cast, arg);
    }   

    public static Expression<Action<T, object>> SetPropertyFunc<T>(PropertyInfo property, Func<Expression, Type, Expression> setterCastExpression)
    {
        //(x, y) => x.Property = (TProperty)y       
        var arg1 = Expression.Parameter(typeof(T), "x");
        var arg2 = Expression.Parameter(typeof(object), "y");
        var cast = setterCastExpression(arg2, property.PropertyType);
        var setter = Expression.Call(arg1, property.GetSetMethod(), cast);
        return Expression.Lambda<Action<T, object>>(setter, arg1, arg2);
    }   
}

Notes:

• CustomType remains as shown in your question.

• No attempt was made to handle the presence of a naming policy in JsonSerializerOptions.PropertyNamingPolicy. You could implement this in TypeWithOptionalsConverter<T> if necessary.

• I added a non-generic interface IHasValue to enable easier access to a boxed Optional<T> during serialization.

Demo fiddle #2 here.

Alternatively, you could stick with Json.NET which supports this at the property and contact level. See:

• Optionally serialize a property based on its runtime value (essentially a duplicate of your question).

• how to dynamic jsonignore according to user authorize?