Concatenating a C# List of byte[]

Question

I am creating several byte arrays that need to be joined together to create one large byte array - i'd prefer not to use byte[]'s at all but have no choice here...

I am adding each one to a List as I create them, so I only have to do the concatenation once I have all the byte[]'s, but my question is, what is the best way of actually doing this?

When I have a list with an unknown number of byte[]'s and I want to concat them all together.

Thanks.

Answer 1

listOfByteArrs.SelectMany(byteArr=>byteArr).ToArray()

The above code will concatenate a sequence of sequences of bytes into one sequence - and store the result in an array.

Though readable, this is not maximally efficient - it's not making use of the fact that you already know the length of the resultant byte array and thus can avoid the dynamically extended .ToArray() implementation that necessarily involves multiple allocations and array-copies. Furthermore, SelectMany is implemented in terms of iterators; this means lots+lots of interface calls which is quite slow. However, for small-ish data-set sizes this is unlikely to matter.

If you need a faster implementation you can do the following:

var output = new byte[listOfByteArrs.Sum(arr=>arr.Length)];
int writeIdx=0;
foreach(var byteArr in listOfByteArrs) {
    byteArr.CopyTo(output, writeIdx);
    writeIdx += byteArr.Length;
}

or as Martinho suggests:

var output = new byte[listOfByteArrs.Sum(arr => arr.Length)];
using(var stream = new MemoryStream(output))
    foreach (var bytes in listOfByteArrs)
        stream.Write(bytes, 0, bytes.Length);

Some timings:

var listOfByteArrs = Enumerable.Range(1,1000)
    .Select(i=>Enumerable.Range(0,i).Select(x=>(byte)x).ToArray()).ToList();

Using the short method to concatenate these 500500 bytes takes 15ms, using the fast method takes 0.5ms on my machine - YMMV, and note that for many applications both are more than fast enough ;-).

Finally, you could replace Array.CopyTo with the static Array.Copy, the low-level Buffer.BlockCopy, or a MemoryStream with a preallocated back buffer - these all perform pretty much identically on my tests (x64 .NET 4.0).

Answer 2

Here's a solution based on Andrew Bezzub's and fejesjoco's answers, pre-allocating all the memory needed up front. This yields Θ(N) memory usage and Θ(N) time (N being the total number of bytes).

byte[] result = new byte[list.Sum(a => a.Length)];
using(var stream = new MemoryStream(result))
{
    foreach (byte[] bytes in list)
    {
        stream.Write(bytes, 0, bytes.Length);
    }
}
return result;