Buffering byte data in C#

Question

My app reads bytes from a TCP socket and needs to buffer them up, so that I can extract messages from them later. Due to the nature of TCP I may get partial or multiple messages in one read, so after each read I would like to inspect the buffer and extract as many full messages as are available.

Therefore I want a class that allows me to do the following:

• append arbitrary byte[] data to it
• inspect the content without consuming it, in particular checking the amount of content and also searching for the existence of a certain byte or bytes
• extract and consume part of the data as a byte[], while leaving the rest in there for a future read

I expect that what I want can be done with 1 or more existing classes in the .NET library, but I'm not sure which ones. System.IO.MemoryStream looks close to what I want, but (a) it isn't clear whether it's suited to being used as a buffer (does the read data get removed from the capacity?) and (b) reads and writes seem to happen at the same place - "The current position of a stream is the position at which the next read or write operation could take place." - which is not what I want. I need to be writing to the end and reading from the front.

Answer 1

I suggest you use MemoryStream under the hood, but encapsulate it in another class which stores:

• The MemoryStream
• The current "read" position
• The current "consumed" position

It will then expose:

• Write: set the stream's position to the end, write data, set the stream's position back to the read position
• Read: read data, set the read position to the stream's position
• Consume: update the consumed position (details based on how you're trying to consume); if the consume position is above a certain threshold, copy the existing buffered data into a new MemoryStream and update all the variables. (You probably don't want to copy the buffer on every consume request.)

Note that none of this will be thread-safe without extra synchronization.

Answer 2

Just use a big byte-array and Array.Copy - it should do the trick. If not, use List<byte>.

If you use the array you have to implement an index to it (where you copy additional data) yourself (same for checking the content-size), but it's straightforward.

If you are interested: here is a simple implementation of a "cyclic buffer". The test should run (I threw a couple unit test at it, but it didn't check all critical path):

public class ReadWriteBuffer
{
    private readonly byte[] _buffer;
    private int _startIndex, _endIndex;

    public ReadWriteBuffer(int capacity)
    {
        _buffer = new byte[capacity];
    }

    public int Count
    {
        get
        {
            if (_endIndex > _startIndex)
                return _endIndex - _startIndex;
            if (_endIndex < _startIndex)
                return (_buffer.Length - _startIndex) + _endIndex;
            return 0;
        }
    }

    public void Write(byte[] data)
    {
        if (Count + data.Length > _buffer.Length)
            throw new Exception("buffer overflow");
        if (_endIndex + data.Length >= _buffer.Length)
        {
            var endLen = _buffer.Length - _endIndex;
            var remainingLen = data.Length - endLen;

            Array.Copy(data, 0, _buffer, _endIndex, endLen);
            Array.Copy(data, endLen, _buffer, 0, remainingLen);
            _endIndex = remainingLen;
        }
        else
        {
            Array.Copy(data, 0, _buffer, _endIndex, data.Length);
            _endIndex += data.Length;
        }
    }

    public byte[] Read(int len, bool keepData = false)
    {
        if (len > Count)
            throw new Exception("not enough data in buffer");
        var result = new byte[len];
        if (_startIndex + len < _buffer.Length)
        {
            Array.Copy(_buffer, _startIndex, result, 0, len);
            if (!keepData)
                _startIndex += len;
            return result;
        }
        else
        {
            var endLen = _buffer.Length - _startIndex;
            var remainingLen = len - endLen;
            Array.Copy(_buffer, _startIndex, result, 0, endLen);
            Array.Copy(_buffer, 0, result, endLen, remainingLen);
            if (!keepData)
                _startIndex = remainingLen;
            return result;
        }
    }

    public byte this[int index]
    {
        get
        {
            if (index >= Count)
                throw new ArgumentOutOfRangeException();
            return _buffer[(_startIndex + index) % _buffer.Length];
        }
    }

    public IEnumerable<byte> Bytes
    {
        get
        {
            for (var i = 0; i < Count; i++)
                yield return _buffer[(_startIndex + i) % _buffer.Length];
        }
    }
}

Please note: the code "consumes" on read - if you don't want that just remove the "_startIndex = ..." parts (or make a overload optional parameter and check or whatever).