TStringList and TThread that does not free all of its memory

Question

Version used: Delphi 7.

I'm working on a program that does a simple for loop on a Virtual ListView. The data is stored in the following record:

type TList=record
  Item:Integer;
  SubItem1:String;
  SubItem2:String;
end;

Item is the index. SubItem1 the status of the operations (success or not). SubItem2 the path to the file. The for loop loads each file, does a few operations and then, save it. The operations take place in a TStringList. Files are about 2mb each.

Now, if I do the operations on the main form, it works perfectly.

Multi-threaded, there is a huge memory problem. Somehow, the TStringList doesn't seem to be freed completely. After 3-4k files, I get an EOutofMemory exception. Sometimes, the software is stuck to 500-600mb, sometimes not. In any case, the TStringList always return an EOutofMemory exception and no file can be loaded anymore. On computers with more memory, it takes longer to get the exception.

The same thing happens with other components. For instance, if I use THTTPSend from Synapse, well, after a while, the software cannot create any new threads because the memory consumption is too high. It's around 500-600mb while it should be, max, 100mb. On the main form, everything works fine.

I guess the mistake is on my side. Maybe I don't understand threads enough. I tried to free everything on the Destroy event. I tried FreeAndNil procedure. I tried with only one thread at a time. I tried freeing the thread manually (no FreeOnTerminate...)

No luck.

So here is the thread code. It's only the basic idea; not the full code with all the operations. If I remove the LoadFile prodecure, everything works good. A thread is created for each file, according to a thread pool.

unit OperationsFiles;

interface

uses Classes, SysUtils, Windows;

type
 TOperationFile = class(TThread)
 private
  Position : Integer;
  TPath, StatusMessage: String;
  FileStringList: TStringList;
  procedure UpdateStatus;
  procedure LoadFile;
 protected
  procedure Execute; override;
 public
  constructor Create(Path: String; LNumber: Integer);
 end;

implementation

uses Form1;

procedure TOperationFile.LoadFile;
begin
 try
  FileStringList.LoadFromFile(TPath);
  // Operations...
  StatusMessage := 'Success';
 except
  on E : Exception do StatusMessage := E.ClassName;
 end;
end;

constructor TOperationFile.Create(Path : String; LNumber: Integer);
begin
 inherited Create(False);
 TPath := Path;
 Position := LNumber;
 FreeOnTerminate := True;
end;

procedure TOperationFile.UpdateStatus;
begin
 FileList[Position].SubItem1 := StatusMessage;
 Form1.ListView4.UpdateItems(Position,Position);
end;

procedure TOperationFile.Execute;
begin
 FileStringList:= TStringList.Create;
 LoadFile;

 Synchronize(UpdateStatus);

 FileStringList.Free;
end;

end.

What could be the problem?

I thought at one point that, maybe, too many threads are created. If a user loads 1 million files, well, ultimately, 1 million threads is going to be created -- although, only 50 threads are created and running at the same time.

Thanks for your input.

Answer 1

There are (probably) no leaks in the code you show in the question.

I say probably because an exception raised during the Execute could result in a leak. The lifetime of the string list should be protected by a finally block.

FileStringList:= TStringList.Create;
try
  LoadFile;
  Synchronize(UpdateStatus);
finally
  FileStringList.Free;
end;

That said, I expect the exception swallow in LoadFile means that you don't leak the string list.

You say that perhaps thousands of threads are created. Each thread reserves memory for its stack, and the default stack size is 1MB. Once you have thousands of 1MB stacks reserved, you can easily exhaust or fragment address space.

I've seen problems due to cavalier creation of threads in the past. For example I had a program that failed when it created and destroyed threads, with never more than 256 threads in existence. This was on a 16 core machine with 4GB address space. You probably have 2GB address space available.

Although you state that no more than 50 threads are in existence at any one moment, I'm not sure how you can be sure of that. Not least, because you have set FreeOnTerminate to True and thereby surrendered control over the lifetime of your threads.

My guess is that your problems are related to the number of threads you create. One thread per processor will suffice. Re-use your threads. It's expensive to create and destroy a thread for a small task.

If this is not enough to solve your problems then you will need to show the code that manages thread lifetime.

Finally, I wonder how much benefit you will extract from threading this app. If it is IO bound then the threaded version may well be slower!