Error converting value from string to stream

Question

I want to serialize a custom object :

public class MyCustomObject
{
    public string Name { get; set; }
    public DateTime Date { get; set; }
    public List<HttpPostedFileBase> Files { get; set; }

    public MyCustomObject()
    {
        Files = new List<HttpPostedFileBase>();
    }
}

In json. To do this, I use a custom converter :

public class HttpPostedFileConverter : JsonConverter
{
    public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
    {
        var stream = (Stream)value;
        using (var sr = new BinaryReader(stream))
        {
            var buffer = sr.ReadBytes((int)stream.Length);
            writer.WriteValue(Convert.ToBase64String(buffer));
        }
    }

I use a JsonSerializerSettings to serialize to json.net knows which type implement (for HttpPostedFileBase).

        var settings = new JsonSerializerSettings();
        settings.Converters.Add(new HttpPostedFileConverter());
        settings.TypeNameHandling = TypeNameHandling.Objects;

The object is serialized correctly but I have this error for the serialization :

JsonSerializationException Error converting value "/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAIBAQEBAQIBAQECAgICAgQDAgICAgUEBAMEBgUGBgYFBgYGBwkIBgcJBwYGCAsICQoKCgoKBggLDAsKDA

and this is the value of my serialized object:

{
  "$type": "ConsoleApplication1.MyCustomObject, ConsoleApplication1",
  "Name": "Test2",
  "Date": "2016-11-03T12:35:14.6020154+01:00",
  "Files": [
    {
      "$type": "System.Web.HttpPostedFileWrapper, System.Web",
      "ContentLength": 1024,
      "FileName": "Pannigale.jpg",
      "ContentType": "image/jpg",
      "InputStream": "/9j/4AAQ...KKAP//Z"
    }
  ]
}

What's wrong in the deserialization ?

EDIT I have tested a class to test... and now it works :

public class TestHttpFile : HttpPostedFileBase
{
    string fullFileName = @"C:\Pictures\SBK-1299-Panigale-S_2015_Studio_R_B01_1920x1080.mediagallery_output_image_[1920x1080].jpg";
    public override int ContentLength
    {
        get
        {
            return 1024;
        }
    }

    public override string FileName
    {
        get
        {
            return "Pannigale.jpg";
        }
    }

    public override string ContentType
    {
        get
        {
            return "image/jpg";
        }
    }

    public override Stream InputStream
    {
        get
        {
            return File.OpenRead(fullFileName);
        }
    }
}

In the serialization I noticed this difference :

 "$type": "ConsoleApplication1.TestHttpFile, ConsoleApplication1",

instead of

 "$type": "System.Web.HttpPostedFileWrapper, System.Web",

But finally I don't want to create a wrapper or whatever... and I don't understand why it works with this type and not with the HttpPostedFileWrapper.

Answer 1

Why Deserialization Fails

In brief: the types you are trying to deserialize, namely HttpPostedFileWrapper and its underlying HttpPostedFile, are not publicly constructible.

Presumably your HttpPostedFileConverter is actually a stream converter, and works something like this:

public class StreamConverter : JsonConverter
{
    public override bool CanConvert(Type objectType)
    {
        return typeof(Stream).IsAssignableFrom(objectType);
    }

    public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
    {
        if (reader.TokenType == JsonToken.Null)
            return null;
        var bytes = serializer.Deserialize<byte[]>(reader);
        return new MemoryStream(bytes);
    }

    public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
    {
        var stream = (Stream)value;
        var bytes = stream.ReadAllBytesAndReposition();
        serializer.Serialize(writer, bytes);
    }
}

public static class StreamExtensions
{
    public static byte[] ReadAllBytesAndReposition(this Stream stream)
    {
        const int bufferSize = 4096;
        using (var ms = new MemoryStream())
        {
            byte[] buffer = new byte[bufferSize];
            int count;
            var position = stream.CanSeek ? stream.Position : (long?)null;
            while ((count = stream.Read(buffer, 0, buffer.Length)) != 0)
                ms.Write(buffer, 0, count);
            if (position != null)
            {
                // Restore position
                stream.Position = position.Value;
            }
            return ms.ToArray();
        }
    }
}

In that case, I can generate JSON similar to yours for your MyCustomObject class, assuming I also set TypeNameHandling = TypeNameHandling.Auto in serializer settings. However, deserialization will fail because the concrete type System.Web.HttpPostedFileWrapper has only one constructor, which takes a HttpPostedFile. From the reference source:

public class HttpPostedFileWrapper : HttpPostedFileBase {

    private HttpPostedFile _file;

    public HttpPostedFileWrapper(HttpPostedFile httpPostedFile) {
        if (httpPostedFile == null) {
            throw new ArgumentNullException("httpPostedFile");
        }
        _file = httpPostedFile;
    }

Since the constructor is public, Json.NET will call it, but as there is no property named httpPostedFile in the JSON it will pass null for that value, causing an ArgumentNullException to be thrown.

This brings us to the next problem: there is no public constructor for HttpPostedFile. Instead, to create one directly you would have to invoke several different Microsoft internal methods via reflection, as is shown in How to instantiate a HttpPostedFile. Thus, even if you created a custom JsonConverter for HttpPostedFileWrapper it is very tricky to construct the requisite HttpPostedFile inside. Json.NET certainly cannot do it automatically.

So, what are your options?

Option 1: Create your own subtype of HttpPostedFilesBase

You can create your own subtype of HttpPostedFilesBase that can be serialized and deserialized successfully, then map all instances of HttpPostedFilesBase to this type during serialization with an appropriate JsonConverter. The following type MemoryHttpPostedFile does this:

public sealed class MemoryHttpPostedFile : HttpPostedFileBase
{
    readonly string contentType;
    readonly string fileName;
    readonly MemoryStream inputStream;

    public MemoryHttpPostedFile(string contentType, string fileName, [JsonConverter(typeof(StreamConverter))] MemoryStream inputStream)
    {
        if (inputStream == null)
            throw new ArgumentNullException("inputStream");
        this.contentType = contentType;
        this.fileName = fileName;
        this.inputStream = inputStream;
    }

    public override int ContentLength { get { return (int)inputStream.Length; } }

    public override string ContentType { get { return contentType; } }

    public override string FileName { get { return fileName; } }

    [JsonConverter(typeof(StreamConverter))]
    public override Stream InputStream { get { return inputStream; } }

    //TODO: implement SaveAs()
    public override void SaveAs(string filename)
    {
        // Implement based on HttpPostedFile.SaveAs()
        // https://referencesource.microsoft.com/#System.Web/HttpPostedFile.cs,678e7f8bc95c149f
        throw new NotImplementedException();
    }
}

public class HttpPostedFileBaseConverter : JsonConverter
{
    public override bool CanConvert(Type objectType)
    {
        return typeof(HttpPostedFileBase).IsAssignableFrom(objectType)
            && !typeof(MemoryHttpPostedFile).IsAssignableFrom(objectType);
    }

    public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
    {
        var postedFile = (HttpPostedFileBase)value;

        // Save position
        var wrapper = new MemoryHttpPostedFile(postedFile.ContentType, postedFile.FileName, new MemoryStream(postedFile.InputStream.ReadAllBytesAndReposition()));

        serializer.Serialize(writer, wrapper);
    }

    public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
    {
        if (reader.TokenType == JsonToken.Null)
            return null;
        var wrapper = serializer.Deserialize<MemoryHttpPostedFile>(reader);
        return wrapper;
    }
}

Since all subtypes of HttpPostedFilesBase get mapped to MemoryHttpPostedFile it is no longer necessary to set TypeNameHandling.Auto. Note I am using StreamConverter and StreamExtensions from earlier in this answer.

Add the converter into settings like so:

var settings = new JsonSerializerSettings
{
    Converters = new JsonConverter[] { new HttpPostedFileBaseConverter() },
    Formatting = Formatting.Indented,
};

Option 2: Actually serialize and deserialize a HttpPostedFileWrapper

To do this we will need to use the rather elaborate and "hacky" code from this answer to How to instantiate a HttpPostedFile by paracycle, like so:

public class HttpPostedFileBaseConverter : JsonConverter
{
    class HttpPostedFileSurrogate
    {
        public string ContentType { get; set; }

        public string FileName { get; set; }

        public byte[] InputStream { get; set; }
    }

    public override bool CanConvert(Type objectType)
    {
        return typeof(HttpPostedFileBase).IsAssignableFrom(objectType);
    }

    public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
    {
        var wrapper = (HttpPostedFileWrapper)value;

        // Save position
        var surrogate = new HttpPostedFileSurrogate
        {
            ContentType = wrapper.ContentType,
            FileName = wrapper.FileName,
            InputStream = wrapper.InputStream.ReadAllBytesAndReposition(),
        };

        serializer.Serialize(writer, surrogate);
    }

    public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
    {
        if (reader.TokenType == JsonToken.Null)
            return null;
        var surrogate = serializer.Deserialize<HttpPostedFileSurrogate>(reader);
        var file = HttpPostedFileExtensions.ConstructHttpPostedFile(surrogate.InputStream, surrogate.FileName, surrogate.ContentType);
        return new HttpPostedFileWrapper(file);
    }
}

public static class HttpPostedFileExtensions
{
    public static HttpPostedFile ConstructHttpPostedFile(byte[] data, string filename, string contentType)
    {
        // Adapted from https://stackoverflow.com/questions/5514715/how-to-instantiate-a-httppostedfile/5515134#5515134
        // Get the System.Web assembly reference (they seem to be in different assemblies in different versions of .Net
        var assemblies = new[] { typeof(HttpPostedFile).Assembly, typeof(HttpPostedFileBase).Assembly };

        // Get the types of the two internal types we need
        Type typeHttpRawUploadedContent = assemblies.Select(a => a.GetType("System.Web.HttpRawUploadedContent")).Where(t => t != null).First();
        Type typeHttpInputStream = assemblies.Select(a => a.GetType("System.Web.HttpInputStream")).Where(t => t != null).First();

        // Prepare the signatures of the constructors we want.
        Type[] uploadedParams = { typeof(int), typeof(int) };
        Type[] streamParams = { typeHttpRawUploadedContent, typeof(int), typeof(int) };
        Type[] parameters = { typeof(string), typeof(string), typeHttpInputStream };

        // Create an HttpRawUploadedContent instance
        object uploadedContent = typeHttpRawUploadedContent
          .GetConstructor(BindingFlags.NonPublic | BindingFlags.Instance, null, uploadedParams, null)
          .Invoke(new object[] { data.Length, data.Length });

        // Call the AddBytes method
        typeHttpRawUploadedContent
          .GetMethod("AddBytes", BindingFlags.NonPublic | BindingFlags.Instance)
          .Invoke(uploadedContent, new object[] { data, 0, data.Length });

        // This is necessary if you will be using the returned content (ie to Save)
        typeHttpRawUploadedContent
          .GetMethod("DoneAddingBytes", BindingFlags.NonPublic | BindingFlags.Instance)
          .Invoke(uploadedContent, null);

        // Create an HttpInputStream instance
        object stream = (Stream)typeHttpInputStream
          .GetConstructor(BindingFlags.NonPublic | BindingFlags.Instance, null, streamParams, null)
          .Invoke(new object[] { uploadedContent, 0, data.Length });

        // Create an HttpPostedFile instance
        HttpPostedFile postedFile = (HttpPostedFile)typeof(HttpPostedFile)
          .GetConstructor(BindingFlags.NonPublic | BindingFlags.Instance, null, parameters, null)
          .Invoke(new object[] { filename, contentType, stream });

        return postedFile;
    }
}

Note I am using StreamExtensions from earlier in this answer. This version of HttpPostedFileBaseConverter would also get added to JsonSerializerSettings.Converters as usual.

(Honestly, I don't recommend this solution, as it relies far too much on the undocumented internals of Microsoft's implementation of HttpPostedFile, which could easily change in later releases.)

Answer 2

Actually there are 2 separate issues.

Stream converter

I want to serialize a custom object ... in json. To do this, I use a custom converter...

Although you've called it HttpPostedFileConverter, the posted code indicates that all it's doing is to convert a Stream to Base64 encoded string. So let call it differently and implement all methods:

public class StreamConverter : JsonConverter
{
    public override bool CanConvert(Type objectType)
    {
        return typeof(Stream).IsAssignableFrom(objectType);
    }

    public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
    {
        return new MemoryStream(Convert.FromBase64String((string)reader.Value));
    }

    public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
    {
        var stream = (Stream)value;
        using (var sr = new BinaryReader(stream))
        {
            var buffer = sr.ReadBytes((int)stream.Length);
            writer.WriteValue(Convert.ToBase64String(buffer));
        }
    }
}

It will be used for InputStream member serialization/deserialization, so in both places you'll use:

settings.Converters.Add(new StreamConverter());

For serialization, that's all you need. Now the tricky part.

Deserializing HttpPostedFileBase instances

But finally I don't want to create a wrapper or whatever... and I don't understand why it works with this type and not with the HttpPostedFileWrapper

The problem is that HttpPostedFileWrapper used for serialization have no parameterless constructor. In fact, the only available constructor has the following signature:

public HttpPostedFileWrapper(
    HttpPostedFile httpPostedFile
)

As you can see, it requires HttpPostedFile. But HttpPostedFile has no public construct at all.

So, want or not, you have no other choice than creating a custom class to be used for deserializing HttpPostedFileBase instances.

Let's do that:

public class CustomHttpPostedFile : HttpPostedFileBase
{
    [JsonProperty("ContentLength")]
    private int contentLength;
    [JsonProperty("FileName")]
    private string fileName;
    [JsonProperty("ContentType")]
    private string contentType;
    [JsonProperty("InputStream")]
    private Stream inputStream;
    [JsonIgnore]
    public override int ContentLength { get { return contentLength; } }
    [JsonIgnore]
    public override string FileName { get { return fileName; } }
    [JsonIgnore]
    public override string ContentType { get { return contentType; } }
    [JsonIgnore]
    public override Stream InputStream { get { return inputStream; } }
    public override void SaveAs(string filename)
    {
        using (var output = File.Create(fileName))
            InputStream.CopyTo(output);
    }
}

But that's not enough. In order to let the deserializer use this class instead of HttpPostedFileWrapper, you'll need a CustomCreationConverter as in Json.NET Deserialize with CustomCreationConverter sample:

public class HttpPostedFileBaseConverter : CustomCreationConverter<HttpPostedFileBase>
{
    public override HttpPostedFileBase Create(Type objectType)
    {
        return new CustomHttpPostedFile();
    }
}

and use it only when deserializing:

settings.Converters.Add(new StreamConverter());
settings.Converters.Add(new HttpPostedFileBaseConverter());

and everything will work.

To recap, the main issue you are experiencing is caused by the usage inside your object of an abstract class and its concrete implementation which you have no control and does not provide a way to be deserialized (because is not designed to be used for serialization at all). So the good thing is that the solution exists at all, with the cost of writing some custom code.