Encoding issue in TStreamReader with plain csv file

Question

I have a problem reading a plain csv file, generated with Excel 2013. It seems the encoding is not working correctly within the TStreamReader class. The strange thing is, one file is working the other one not. When reading the second file, TStreamReader returns an empty string:

LString := FEncoding.GetString(LBuffer, StartIndex, ByteBufLen);

Both files have the 1 Byte ANSI encoding. But TStreamReader is using UTF8 encoding.

My code:

  fs := TFileStream.Create(aFileName, fmOpenRead or fmShareDenyNone);
  sr := TStreamReader.Create(fs);
  while (not sr.EndOfStream) do //sr.EndOfStream is always true!!!!
  begin
    //some code here
  end;

So far I figured out, that the following function is returning an empty string:

function TMBCSEncoding.GetCharCount(Bytes: PByte; ByteCount: Integer): Integer;
begin
  Result := UnicodeFromLocaleChars(FCodePage, FMBToWCharFlags,
    PAnsiChar(Bytes), ByteCount, nil, 0);
end;

When I compare both files, they have the same inputs beside the Bytes and ByteCount variable. But the Bytes startes with same values (same csv header names).

So my question is, why is one file working and the other not? What can I do to read the files correctly?

Answer 1

The constructor for TStreamReader that you call is this one:

constructor TStreamReader.Create(Stream: TStream);
begin
  Create(Stream, TEncoding.UTF8, True);
end;

The True argument is DetectBOM. If a BOM is encountered, that will determine the encoding. Otherwise the file will be treated as UTF-8. Your files don't have BOMs. Therefore you are getting exactly what you asked for. Namely that the file is treated as UTF-8.

If you want the file treated as ANSI you must specify the encoding:

sr := TStreamReader.Create(fs, TEncoding.Default);

Or if you want to default to ANSI if no BOM is found, otherwise respect the BOM, you can do it like this:

sr := TStreamReader.Create(fs, TEncoding.Default, True);

Why does your code work with one file but not the other? Presumably one file is entirely in the ASCII range, and the other has characters outside that range. UTF-8 encodes characters in the ASCII range in a single byte which means that ASCII encoded files are correctly interpreted by the UTF-8 encoding. That was one of the primary design goals of UTF-8.