Can't retrieve TStreams bigger than around 260.000 bytes from a Datasnap Server

Question

I have a Delphi 10.1 Berlin Datasnap Server, that can't return Data packets (through a TStream) bigger than around 260.000 bytes.

I have programmed it following the \Object Pascal\DataSnap\FireDAC sample from Delphi, which also shows this problem.

The problem can be seen just opening that sample, setting blank the IndexFieldName of the qOrders component on ServerMethodsUnit.pas, and changing its SQL property to :

select * from Orders
union 
select * from Orders

Now the amount of data to be send is beyond 260.000 bytes, which seems to be the point where you can't retrieve it from the client. Getting a EFDException [FireDAC][Stan]-710. Invalid binary storage format.

The data is sent as a Stream that you get from a FDSchemaAdapter on the server, and you load on another FDSchemaAdpater on the client. The connection between Client and Server is also FireDAC.

This is how the Server returns that Stream :

function TServerMethods.StreamGet: TStream;
begin
  Result := TMemoryStream.Create;
  try
    qCustomers.Close;
    qCustomers.Open;
    qOrders.Close;
    qOrders.Open;
    FDSchemaAdapter.SaveToStream(Result, TFDStorageFormat.sfBinary);
    Result.Position := 0;
  except
    raise;
  end;
end;

And this is how the Client retrieves it :

procedure TClientForm.GetTables;
var
  LStringStream: TStringStream;
begin
  FDStoredProcGet.ExecProc;
  LStringStream := TStringStream.Create(FDStoredProcGet.Params[0].asBlob);
  try
    if LStringStream <> nil then
    begin
      LStringStream.Position := 0;
      DataModuleFDClient.FDSchemaAdapter.LoadFromStream(LStringStream, TFDStorageFormat.sfBinary);
    end;
  finally
    LStringStream.Free;
  end;
end;

The Client doesn't get all the data on the Blob parameter. I save the content of the Stream on the Server, and the content that arrives at the Blob parameter on the Client, and they have the same size, but the content of the Blob parameter has its content truncated, and the last few Kbytes are zeroes.

This is how I save on the Server the content that will go to the Stream:

FDSchemaAdapter.SaveToFile('C:\Temp\JSON_Server.json', TFDStorageFormat.sfJSON);

This is how I check what I get on the Client blob parameter:

TFile.WriteAllText('C:\Temp\JSON_Client.json', FDStoredProcGet.Params[0].asBlob);

I can see that the Client gets the data truncated.

Do you know how to fix it, or a workaround to retrieve all the Stream content from the Datasnap Server to my Client ?.

Update: I have updated to Delphi 10.1 Berlin Update 2, but the problem remains.

Thank you.

Answer 1

I get a similar problem with Seattle (I don't have Berlin installed) with a DataSnap server that doesn't involve FireDAC.

On my DataSnap server I have:

type
  TServerMethods1 = class(TDSServerModule)
  public
    function GetStream(Size: Integer): TStream;
    function GetString(Size: Integer): String;
  end;

[...]

uses System.StrUtils;

function BuildString(Size : Integer) : String;
var
  S : String;
  Count,
  LeftToWrite : Integer;
const
  scBlock = '%8d bytes'#13#10;
begin
  LeftToWrite := Size;
  Count := 1;
  while Count <= Size do begin
    S := Format(scBlock, [Count]);
    if LeftToWrite >= Length(S) then
    else
      S := Copy(S, 1, LeftToWrite);
    Result := Result + S;
    Inc(Count, Length(S));
    Dec(LeftToWrite, Length(S));
  end;
  if Length(Result) > 0 then
    Result[Length(Result)] := '.'
end;

function TServerMethods1.GetStream(Size : Integer): TStream;
var
  SS : TStringStream;
begin
  SS := TStringStream.Create;
  SS.WriteString(BuildString(Size));
  SS.Position := 0;
  OutputDebugString('Quality Suite:TRACING:ON');
  Result := SS;
end;

function TServerMethods1.GetString(Size : Integer): String;
begin
  Result := BuildString(Size);
end;

As you can see, both these functions build a string of the specified size using the same BuildString function and return it as a stream and a string respectively.

On two Win10 systems here, GetStream works fine for sizes up to 30716 bytes but above that, it returns an empty stream and a "size" of -1.

Otoh, GetString works fine for all sizes I have tested up to and including a size of 32000000. I have not yet managed to trace why GetStream fails. However, based on the observation that GetString does work, I tested the following work-around, which sends a stream as a string, and that works fine up to 32M as well:

function TServerMethods1.GetStreamAsString(Size: Integer): String;
var
  S : TStream;
  SS : TStringStream;
begin
  S := GetStream(Size);
  S.Position := 0;
  SS := TStringStream.Create;
  SS.CopyFrom(S, S.Size);
  SS.Position := 0;
  Result := SS.DataString;
  SS.Free;
  S.Free;
end;

I appreciate you might prefer your own work-around of sending the result in chunks.

Btw, I have tried calling my GetStream on the server by creating an instance of TServerMethods in a method of the server's main form and calling GetStream directly from that, so that the server's TDSTCPServerTransport isn't involved. This correctly returns the stream so the problem seems to be in the transport layer or the input and/or output interfaces to it.