Is using TStringList to load huge text file the best way in Delphi?

Question

What is the best way to load huge text file data in delphi? Is there any component that can load text file superfast?

Let's say I have a text file contains database and stored in fix length format. It contains 150 field with each at least 50 characters. 1. I need to load it into memory 2. I need to parse it and probably store it in a memdataset for processing

My questions: 1. Is it enough if I use TStringList.loadFromFile method? 2. Is there any other better component to manipulate the text file? 3. Should I use low level reading from textfile?

Thank you in advance.

Answer 1

TStringList is never the optimal way of working with lots of text, but it's the simplest. If you've got small files on your hands you can use TStringList without issues. Even if you have large files (not huge files) you might implement a version of you algorithm using TStringList for testing purposes, because it's simple and easy to understand.

If your files are large, as they probably are since you call them "databases", you need to look into alternative technologies that will enable you to read only as much as you need from the database. Look into:

• TFileStream
• Memory mapped files.

Don't look at the old "file" based API's still available in Delphi, they're plain old.

I'm not going to go into details on how to access text using those methods because we've recently had two similar questions on SO:

How Can I Efficiently Read The FIrst Few Lines of Many Files in Delphi

and

Fast Search to see if a String Exists in Large Files with Delphi