Serializing a ConcurrentBag of XAML

Question

I have, in my code, a ConcurrentBag<Point3DCollection>.

I'm trying to figure out how to serialize them. Of course I could iterate through or package it with a provider model class, but I wonder if it's already been done.

The Point3DCollections themselves are potentially quite large and could stand to be compressed to speed up reading and writing to and from the disk, but the response times I need for this are largely in the user interface scale. In other words, I prefer a binary formatting over a XAML-text formatting, for performance reasons. (There is a nice XAML-text serializer which is part of the Helix 3D CodeProject, but it's slower than I'd like.)

Is this a use case where I'm left rolling out my own serializer, or is there something out there that's already packaged for this kind of data?

Answer 1

Here are some extensions methods that handle string and binary serialization of Point3DCollection bags. As I said in my comment, I don't think there is a best way of doing this in all cases, so you might want to try both. Also note they're using Stream parameter as input so you can chain these with calls to GZipStream of DeflateStream.

public static class Point3DExtensions
{
    public static void StringSerialize(this ConcurrentBag<Point3DCollection> bag, Stream stream)
    {
        if (bag == null)
            throw new ArgumentNullException("bag");

        if (stream == null)
            throw new ArgumentNullException("stream");

        StreamWriter writer = new StreamWriter(stream);
        Point3DCollectionConverter converter = new Point3DCollectionConverter();
        foreach (Point3DCollection coll in bag)
        {
            // we need to use the english locale as the converter needs that for parsing...
            string line = (string)converter.ConvertTo(null, CultureInfo.GetCultureInfo("en-US"), coll, typeof(string));
            writer.WriteLine(line);
        }
        writer.Flush();
    }

    public static void StringDeserialize(this ConcurrentBag<Point3DCollection> bag, Stream stream)
    {
        if (bag == null)
            throw new ArgumentNullException("bag");

        if (stream == null)
            throw new ArgumentNullException("stream");

        StreamReader reader = new StreamReader(stream);
        Point3DCollectionConverter converter = new Point3DCollectionConverter();
        do
        {
            string line = reader.ReadLine();
            if (line == null)
                break;

            bag.Add((Point3DCollection)converter.ConvertFrom(line));

            // NOTE: could also use this:
            //bag.Add(Point3DCollection.Parse(line));
        }
        while (true);
    }

    public static void BinarySerialize(this ConcurrentBag<Point3DCollection> bag, Stream stream)
    {
        if (bag == null)
            throw new ArgumentNullException("bag");

        if (stream == null)
            throw new ArgumentNullException("stream");

        BinaryWriter writer = new BinaryWriter(stream);
        writer.Write(bag.Count);
        foreach (Point3DCollection coll in bag)
        {
            writer.Write(coll.Count);
            foreach (Point3D point in coll)
            {
                writer.Write(point.X);
                writer.Write(point.Y);
                writer.Write(point.Z);
            }
        }
        writer.Flush();
    }

    public static void BinaryDeserialize(this ConcurrentBag<Point3DCollection> bag, Stream stream)
    {
        if (bag == null)
            throw new ArgumentNullException("bag");

        if (stream == null)
            throw new ArgumentNullException("stream");

        BinaryReader reader = new BinaryReader(stream);
        int count = reader.ReadInt32();
        for (int i = 0; i < count; i++)
        {
            int pointCount = reader.ReadInt32();
            Point3DCollection coll = new Point3DCollection(pointCount);
            for (int j = 0; j < pointCount; j++)
            {
                coll.Add(new Point3D(reader.ReadDouble(), reader.ReadDouble(), reader.ReadDouble()));
            }
            bag.Add(coll);
        }
    }
}

And a little console app test program to play with:

    static void Main(string[] args)
    {
        Random rand = new Random(Environment.TickCount);
        ConcurrentBag<Point3DCollection> bag = new ConcurrentBag<Point3DCollection>();
        for (int i = 0; i < 100; i++)
        {
            Point3DCollection coll = new Point3DCollection();
            bag.Add(coll);

            for (int j = rand.Next(10); j < rand.Next(100); j++)
            {
                Point3D point = new Point3D(rand.NextDouble(), rand.NextDouble(), rand.NextDouble());
                coll.Add(point);
            }
        }

        using (FileStream stream = new FileStream("test.bin", FileMode.Create))
        {
            bag.StringSerialize(stream); // or Binary
        }

        ConcurrentBag<Point3DCollection> newbag = new ConcurrentBag<Point3DCollection>();
        using (FileStream stream = new FileStream("test.bin", FileMode.Open))
        {
            newbag.StringDeserialize(stream); // or Binary
            foreach (Point3DCollection coll in newbag)
            {
                foreach (Point3D point in coll)
                {
                    Console.WriteLine(point);
                }
                Console.WriteLine();
            }
        }
    }
}

Answer 2

Compression could potentially take advantage of repeated coordinates. Serializers will often use references for repeat objects as well, although I'm not sure there are many set up to work with structs (like Point3D). Anyhow, here are some examples of how to serialize this. To use the standard formatters, you need to convert the data type to something most of them support: list/array. The code below uses Nuget packages NUnit and Json.NET.

using Newtonsoft.Json;
using Newtonsoft.Json.Bson;
using NUnit.Framework;
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.IO.Compression;
using System.Linq;
using System.Runtime.Serialization.Formatters.Binary;
using System.Text;
using System.Windows.Media.Media3D;

namespace DemoPoint3DSerialize
{
    [TestFixture]
    class Tests
    {
        [Test]
        public void DemoBinary()
        {
            // this shows how to convert them all to strings
            var collection = CreateCollection();
            var data = collection.Select(c => c.ToArray()).ToList(); // switch to serializable types
            var formatter = new BinaryFormatter();

            using (var ms = new MemoryStream())
            {
                formatter.Serialize(ms, data);
                Trace.WriteLine("Binary of Array Size: " + ms.Position);
                ms.Position = 0;
                var dupe = (List<Point3D[]>)formatter.Deserialize(ms);
                var result = new ConcurrentBag<Point3DCollection>(dupe.Select(r => new Point3DCollection(r)));
                VerifyEquality(collection, result);
            }
        }

        [Test]
        public void DemoString()
        {
            // this shows how to convert them all to strings
            var collection = CreateCollection();
            IEnumerable<IList<Point3D>> tmp = collection;
            var strings = collection.Select(c => c.ToString()).ToList();

            Trace.WriteLine("String Size: " + strings.Sum(s => s.Length)); // eh, 2x for Unicode
            var result = new ConcurrentBag<Point3DCollection>(strings.Select(r => Point3DCollection.Parse(r)));

            VerifyEquality(collection, result);
        }

        [Test]
        public void DemoDeflateString()
        {
            // this shows how to convert them all to strings
            var collection = CreateCollection();
            var formatter = new BinaryFormatter(); // not really helping much: could 
            var strings = collection.Select(c => c.ToString()).ToList();

            using (var ms = new MemoryStream())
            {
                using (var def = new DeflateStream(ms, CompressionLevel.Optimal, true))
                {
                    formatter.Serialize(def, strings);
                }
                Trace.WriteLine("Deflate Size: " + ms.Position);
                ms.Position = 0;
                using (var def = new DeflateStream(ms, CompressionMode.Decompress))
                {
                    var stringsDupe = (IList<string>)formatter.Deserialize(def);
                    var result = new ConcurrentBag<Point3DCollection>(stringsDupe.Select(r => Point3DCollection.Parse(r)));

                    VerifyEquality(collection, result);
                }
            }
        }

        [Test]
        public void DemoStraightJson()
        {
            // this uses Json.NET
            var collection = CreateCollection();
            var formatter = new JsonSerializer();

            using (var ms = new MemoryStream())
            {
                using (var stream = new StreamWriter(ms, new UTF8Encoding(true), 2048, true))
                using (var writer = new JsonTextWriter(stream))
                {
                    formatter.Serialize(writer, collection);
                }
                Trace.WriteLine("JSON Size: " + ms.Position);
                ms.Position = 0;
                using (var stream = new StreamReader(ms))
                using (var reader = new JsonTextReader(stream))
                {
                    var result = formatter.Deserialize<List<Point3DCollection>>(reader);
                    VerifyEquality(collection, new ConcurrentBag<Point3DCollection>(result));
                }
            }
        }

        [Test]
        public void DemoBsonOfArray()
        {
            // this uses Json.NET
            var collection = CreateCollection();
            var formatter = new JsonSerializer();

            using (var ms = new MemoryStream())
            {
                using (var stream = new BinaryWriter(ms, new UTF8Encoding(true), true))
                using (var writer = new BsonWriter(stream))
                {
                    formatter.Serialize(writer, collection);
                }
                Trace.WriteLine("BSON Size: " + ms.Position);
                ms.Position = 0;
                using (var stream = new BinaryReader(ms))
                using (var reader = new BsonReader(stream, true, DateTimeKind.Unspecified))
                {
                    var result = formatter.Deserialize<List<Point3DCollection>>(reader); // doesn't seem to read out that concurrentBag
                    VerifyEquality(collection, new ConcurrentBag<Point3DCollection>(result));
                }
            }
        }

        private ConcurrentBag<Point3DCollection> CreateCollection()
        {
            var rand = new Random(42);
            var bag = new ConcurrentBag<Point3DCollection>();

            for (int i = 0; i < 10; i++)
            {
                var collection = new Point3DCollection();
                for (int j = 0; j < i + 10; j++)
                {
                    var point = new Point3D(rand.NextDouble(), rand.NextDouble(), rand.NextDouble());
                    collection.Add(point);
                }
                bag.Add(collection);
            }
            return bag;
        }

        private class CollectionComparer : IEqualityComparer<Point3DCollection>
        {
            public bool Equals(Point3DCollection x, Point3DCollection y)
            {
                return x.SequenceEqual(y);
            }

            public int GetHashCode(Point3DCollection obj)
            {
                return obj.GetHashCode();
            }
        }

        private void VerifyEquality(ConcurrentBag<Point3DCollection> collection, ConcurrentBag<Point3DCollection> result)
        {
            var first = collection.OrderBy(c => c.Count);
            var second = collection.OrderBy(c => c.Count);
            first.SequenceEqual(second, new CollectionComparer());
        }


    }
}

Answer 3

Use Google's protobuf-net. protobuf-net is an open source .net implementation of Google's protocol buffer binary serialization format which can be used as a replacement for the BinaryFormatter serializer. It is probably going to be the fastest solution and easiest to implement.

Here is a link to the the main google wiki for protobuf-net. On the left, you'll find the downloads for all of the most updated binaries.

https://code.google.com/p/protobuf-net/

Here is a great article that you might want to look at first to get a feel for how it works.

http://wallaceturner.com/serialization-with-protobuf-net

Here is a link to a discussion on google's wiki about your specific problem. The answer is at the bottom of the page. That's where I got the code below and substituted with details from your post.

https://code.google.com/p/protobuf-net/issues/detail?id=354

I haven't used it myself but it looks like a very good solution to your stated needs. From what I gather, your code would end up some variation of this.

[ProtoContract]
public class MyClass {
    public ConcurrentQueue<Point3DCollection> Points {get;set;}

    [ProtoMember(1)]
    private Point3DCollection[] Items
    {
        get { return Points.ToArray(); }
        set { Items = new ConcurrentBag<Point3DCollection>(value); }
    }
}

I wish you the best of luck. Take care.