How to choose a file format?

Question

I created a .NET application years ago without thinking too hard about the file format: it uses a soap formatter to serialize our large hierarchy of objects. It was dirt simple to do, and so I didn't give it much thought.

I'm now trying to come up with a more optimal file format considering the following issue: When a file is saved, it ends up being converted to byte array and getting sent over the wire to a database for storage. This ends up being a big problem because you have all your objects in memory, then you allocate more memory for the serializer, and then you allocate even more memory for the byte array. Even modestly sized object graphs end up using a lot of memory to take care of saving the file.

I'm not sure how to improve this both from an file format perspective but also potentially from the perspective of the algorithm (objects -> stream -> byte array)

UPDATE: I'd always been zipping the byte array before sending it over the wire, so while that's good advice, it was already implemented in my application.

I did convert from Soap to Binary Serialization, and that has made a huge difference: our files are about 7x smaller than before. (Your mileage may vary, of course).

Answer 1

If you need efficient serialization, and don't care if its serialized as a binary format, just use standard binary serialization in .NET. You can just decorate your serializable types with the [Serializable] attribute, and use the BinaryFormatter to serialize your objects to byte[].

Answer 2

BinaryFormatter + DeflateStream = Compressed Persisted Objects

using System;
using System.IO;
using System.IO.Compression;
using System.Runtime.Serialization.Formatters.Binary;

namespace CompressedSerialized
{
    class Program
    {
        static void Main(string[] args)
        {
            var obj1 = new MyObject() { Prop1 = "p1", Prop2 = "p2" };
            MyObject obj2 = null;
            var bin = new BinaryFormatter();
            byte[] buffer = null;

            using (var ms = new MemoryStream())
            {
                using (var zip = new DeflateStream(ms, CompressionMode.Compress))
                {
                    bin.Serialize(zip, obj1);
                    zip.Flush();
                }
                buffer = ms.ToArray();
            }

            using (var ms = new MemoryStream(buffer))
            using (var unzip = new DeflateStream(ms, CompressionMode.Decompress))
            {
                var des = bin.Deserialize(unzip);
                obj2 = des as MyObject;
            }

        }
    }

    [Serializable]
    public class MyObject
    {
        public string Prop1 { get; set; }
        public string Prop2 { get; set; }
    }
}