How to deserialize stream to object using System.Text.Json APIs

Question

I'm receiving a response from a web api call as a stream and need to deserialize it to a model.

This is a generic method, so I can't say which parts of code will use this and what's the response payload.

Here's the method:

public async Task<T> InvokeAsync<T>(string method)
{
    Stream response = await this.httpClientWrapper.InvokeAsync(method);
    var serializer = new JsonSerializer();
    using var streamReader = new StreamReader(response);
    using var reader = new JsonTextReader(streamReader);
    return serializer.Deserialize<T>(reader);
}

I'm trying to remove Newtonsoft and use System.Text.Json API.

I found this porting guide in corefx repo in Github, where section Reading from a Stream/String states:

We currently (as of .NET Core 3.0 preview 2) do not have a convenient API to read JSON from a stream directly (either synchronously or asynchronously). For synchronous reading (especially of small payloads), you could read the JSON payload till the end of the stream into a byte array and pass that into the reader

So following this advise I come up with the following:

public async Task<T> InvokeAsync<T>(string method)
{
    Stream response = await this.httpClientWrapper.InvokeAsync(method);
    var length = response.Length;
    var buffer = ArrayPool<byte>.Shared.Rent((int)length);
    var memory = new Memory<byte>(buffer);
    await response.WriteAsync(memory);
    var result = JsonSerializer.Deserialize<T>(memory.Span);
    ArrayPool<byte>.Shared.Return(buffer);
    return result;
}

So my question is - did I understand the advise correctly and this is the way to go ?

This implementation probably can be improved on many aspects, but what bothers me most is renting the byte array from the pool e.g. Stream.Length is a long and I convert it to int which can cause OverflowException.

I tried to look into System.IO.Pipelines and use ReadOnlySequence<byte> overloads of JSON API, but it gets very complicated.

Answer 1

I believe that documentation needs to be updated because .NET Core 3 has a method to read from a stream directly. Using it is straight-forward, assuming the stream is encoded in UTF8:

private static readonly JsonSerializerOptions Options = new JsonSerializerOptions();

private static async Task<T> Deserialize<T>(HttpResponseMessage response)
{
    var contentStream = await response.Content.ReadAsStreamAsync();
    var result = await JsonSerializer.DeserializeAsync<T>(contentStream, Options);
    return result;
}

One thing to watch out for is that by default HttpClient will buffer the response content in-memory before returning unless you set the HttpCompletionOption to ResponseHeadersRead when invoking SendAsync:

var response = await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, token);