Optimal Buffer Stream write process

Question

We have our own data streaming algorithm that include some metadata+records+fields values.

Currently we use a TStream and write to add values to the stream. Now I'm wondering if this time cosuming operation could be made faster by using some technic.

Edit: We are only appending data to the end, not moving or seeking.

Some things I was thinking of are:

• Not use Streams buf some large memory allocated buffer to copy data to, the problem is if we go beyond the buffer size, then we have to relocate to some new memory space.
• Use a Stream prefilled with #0s to some size and then start adding values. The rationale is that Tstream have to allocate it's own buffer every time I do a write (I don't know how it really works, just wondering...)

We are adding strings to the TStream and binary data in the form #0#0#0#1 for example.

The data is then transfered via TCP, so it's not about write to a File.

So what is the best method for this?

Answer 1

First, you are assuming the TStream is the bottleneck. You need to profile your code, such as with AQTime, to determine where the bottleneck really is. Don't make assumptions.

Second, what type of TStream are you actually using? TMemoryStream? TFileStream? Something else? Different stream types handle memory differently. TMemoryStream allocates a memory buffer and grows it by a pre-set amount of bytes whenever the buffer fills up. TFileStream, on the other hand, doesn't use any memory at all, it just writes directly to the file and lets the OS handle any buffering.

Regardless of which type of stream you use, one thing you could try is implement your own custom TStream class that has an internal fixed-size buffer and a pointer to your real destination TStream object. You can then pass an instance of your custom class to your algorithm. Have your class override the TStream::Write() method to copy the input data into its buffer until it fills up, then you can Write() the buffer to the destination TStream and clear the buffer. Your algorithm will never know the difference. Both TMemoryStream and TFileStream would benefit from the extra buffering - fewer larger writes means more efficient memory allocations and file I/O. For example:

type
  TMyBufferedStreamWriter = class(TStream)
  private
    fDest: TStream;
    fBuffer: array[0..4095] of Byte;
    fOffset: Cardinal;
  public
    constructor Create(ADest: TStream);
    function Read(var Buffer; Count: Longint): Longint; override;
    function Write(const Buffer; Count: Longint): Longint; override;
    procedure FlushBuffer;
  end;

.

uses
  RTLConsts;

constructor TMyBufferedStreamWriter.Create(ADest: TStream);
begin
  fDest := ADest;
end;

function TMyBufferedStreamWriter.Read(var Buffer; Count: Longint): Longint;
begin
  Result := 0;
end;

function TMyBufferedStreamWriter.Write(const Buffer; Count: Longint): Longint;
var
  pBuffer: PByte;
  Num: Cardinal;
begin
  Result := 0;
  pBuffer := PByte(@Buffer);
  while Count > 0 do
  begin
    Num := Min(SizeOf(fBuffer) - fOffset, Cardinal(Count));
    if Num = 0 then FlushBuffer;
    Move(pBuffer^, fBuffer[fOffset], Num);
    Inc(fOffset, Num);
    Inc(pBuffer, Num);
    Dec(Count, Num);
    Inc(Result, Num);
  end;
end;

procedure TMyBufferedStreamWriter.FlushBuffer;
var
  Idx: Cardinal;
  Written: Longint;
begin
  if fOffset = 0 then Exit;
  Idx := 0;
  repeat
    Written := fDest.Write(fBuffer[Idx], fOffset - Idx);
    if Written < 1 then raise EWriteError.CreateRes(@SWriteError);
    Inc(Idx, Written);
  until Idx = fOffset;
  fOffset := 0;
end;

.

Writer := TMyBufferedStreamWriter.Create(RealStreamHere);
try
  ... write data to Writer normally as needed...
  Writer.FlushBuffer;
finally
  Writer.Free;
end;