Effectively using Delphi to read unknown-sized blocks from a file

Question

In the past, I have seen this work, but I never really understood how it should be done.
Assume we have a file of known data types, but unknown length, like a dynamic array of TSomething, where

type
  TSomething = class
    Name: String;
    Var1: Integer;
    Var2: boolean;
  end;

The problem, though, is that this object type may be extended in the future, adding more variables (e.g. Var3: String).
Then, files saved with an older version will not contain the newest variables.
The File Read procedure should somehow recognize data in blocks, with an algorithm like:

procedure Read(Path: String)
begin
  // Read Array Size
  //   Read TSomething --> where does this record end? May not contain Var3!
  //   --> how to know that the next data block I read is not a new object?
end;

I have seen this work with BlockRead and BlockWrite, and I assume each object should probably write its size before writing itself in the file, but I would appreciate an example (not necessarily code), to know that I am thinking towards the right direction.

Related readings I have found:
SO - Delphi 2010: How to save a whole record to a file?
Delphi Basics - BlockRead
SO - Reading/writing dynamic arrays of objects to a file - Delphi
SO - How Can I Save a Dynamic Array to a FileStream in Delphi?

Answer 1

In order to make this work, you need to write the element size to the file. Then when you read the file, you read that element length which allows you to read each entire element, even if your program does not know how to understand all of it.

In terms of matching up your record with the on-disk record that's easy enough if your record only contains simple types. In that scenario you can read from the file Min(ElementLength, YourRecordSize) bytes into your record.

But it does not look as though you actually have that scenario. Your record is in fact a class and so not suitable for memory copying. What's more, its first member is a string which is most definitely not a simple type.

Back in the day (say the 1970s), the techniques you described were how files were read. But these days programming has moved on. Saving structured data to files usually means using a more flexible and adaptable serialization format. You should be looking to using JSON, XML, YAML or similar for such tasks.

Answer 2

I'd say you need a method of versioning you file. That way you know what version of the record is contained in the file. Write it at the start of the file and then on reading, read in the version identifier first and then use the corresponding structure to read the rest.