Attaching to datasets previously created by another thread

Question

this is a question similar to this one but with a different background.

Compiler: Delphi 2010, soon Delphi XE5.

I have built a nice application that manages data on a remote MySQL server through the ZEOS components. Since the connection could fail and SQL be slow I have used the neat OmniThreadLibrary to create an SQL server watchdog and to offload a lot of "read only" tables loading to threads. As of now I have three data modules manually created before the main form shows, each with their independent TZConnection and some TZReadOnlyQuery components linked to the same data module TZConnection. Each thread instantiates its related data module from within itself and then executes the queries.

The watchdog is working quite well, but I have some doubts about the second part, that is the "read only" tables thread. The queries work already but I have yet to use their results in the main application business code, where I have to insert and update data on other tables.

In my plans I get all these "read only" dataset read and loaded before the main application even connects to them (the whole inter-thread state machine is done already). In theory there should be no concurrency issue as the "read only" tables thread has finished its task and is now being idle. But I don't know what happens if at this point I connect a control or another dataset / datasource / whatever from the main form to the idle-threaded data module.

Will I mess up because the main form TZSession is not the same of the threaded data module? Will I get rare & nasty access violations to be discovered ONLY after having delivered the application (of course!). Basically with what kind of confidence or precautions should I access a query component created in another thread, assuming only the main application does it and only for reading data? Is it even possible / healthy at all? Or am I missing some "best practices" way of doing it?

Thanks in advance.

Answer 1

I am going to post how I have done it. It'll be "terse" (it's still a monumental wall of text!) due to lack of time, if you find something being too obscure feel free to ask. I don't pretend to have written neither the best nor the quickest nor the best formatted code. Just use it as your starting point.

Short recap of the problem domain: having a software that "pre-opens" tables at startup using a thread. This lets the software stay responsive and even perform other non database related startup tasks. I have tested this program for 1 month now and the database components used are enough to prove it's not just a "toy demo" ready to break as you add your 3rd dataset.

Ingredients:

As stated above: Delphi 2010+ (I am now running this in RAD Studio XE5 Ultimate) but could probably work with earlier versions. I just did not test them.

• ZEOS library, 7 and upwards should work including the latest 7.2 alpha.
• OmniThreadLibrary
• In my case I also used JVCL but that's only because I needed some specific data source events the basic components don't offer.

IDE, non code portion

• create 2 data modules that will host the database components: one is for the "preload", threaded portion, the other for the "runtime", read write portion that will be basically the one used by the program users to perform their tasks.

• create 1 unit where to store the thread worker code

• create 1 or more form(s) that will constitute your application.

The two data modules would look like this:

Preload module, to be managed by the worker thread.

Portion of the main database module. It includes several additional datasets that the preload module cannot preload. Those are typically "dynamic" datasets whose queries are directly affected by interaction with the users. The main database module's components started as copy and paste of the preload module so it does not take twice as long to prepare both.

Both the preload and the main database modules come with a ConnectToDatabase and DisconnectFromDatabase procedures that perform all the steps required to have the system up and running.

Very important!

• The preload module performs the "true" queries in a separate thread and fills in their related TClientDataSets. Its components have no events attached. I only use them as blind "static" data containers.
• The main database module will just "attach" to the preload module components.

In example: whereas the preload module cdsProducts ClientDataSet performs a "true database query" with the

cdsProduct => dspProduct => qryProduct

chain, the main database module cdsProduct just takes the preload module's cdsProduct data without performing any query at all (otherwise what'd be the point, performing queries twice?).

You see how counter-intuitively enough, the main database module cdsProduct comes with a linked TDataSetProvider and query components as well. Why? Because I use them to write modified data back.

That is, we have three program phases:

1. Startup, where the preload data module performs the queries (in a thread) and that's it. No events managed, all read only.

2. Running-start phase, where the main database module (in the VCL thread) copies the data gathered in 1 into its ClientDataSets.

3. Running phase, where the users interact with the main database module's ClientDataSets. When they need to save data (and only then), the main database module's DataSetProviders and queries are engaged. They only write.

I could have skipped the whole ClientDataSet => Provider => Query chain for some of those ClientDataSets but most of them require some huge data processing, must update many joined tables by hand and so on, so I just used the full stack.

Code portion

Lets get to some more nitty-gritty details. I can't post the whole stuff since it's a commercial application so I'll only paste some significant snippets.

Threaded, preload data module

procedure TModDBPreload.ConnectToDatabase;
begin
    dbcEShop.Connect;
    SendStatusMessage('Loading languages archive');
    qryLanguage.Open;
    qryLanguage.First;
    SearchOptions := [loCaseInsensitive];
    ModApplicationCommon.ApplicationLocaleInfo.Lock;

    ...

    try
        ...

         // All the queries parameters needing a language id need to be assigned to the locked LocaleInfo object
        qryGeoZone.Params.ParamByName('language_id').AsInteger := ModApplicationCommon.ApplicationLocaleInfo.LocaleIDForQueries;
        cdsGeoZones.Params.ParamByName('language_id').AsInteger := ModApplicationCommon.ApplicationLocaleInfo.LocaleIDForQueries;

        ...

    finally
        ModApplicationCommon.ApplicationLocaleInfo.Unlock;
    end;

  SendStatusMessage('Loading countries archive');
  cdsGeoZones.Open;
      cdsGeoZones.First;
  SendStatusMessage('Loading currencies archive');
  qryCurrency.Open;
  qryCurrency.First;
  Sleep(100);
  SendStatusMessage('Loading products archive');
  cdsProduct.Open;
  cdsProduct.First;
  ...
end;

The above snippet could use a lot of explanations. In particular:

SendStatusMessage('Loading languages archive');

is a thread sending an end user friendly update string to be shown on a status line. Of course the status line is managed by the main VCL thread. How to do it? I'll show it later.

qryLanguage.Open;
qryLanguage.First;
...
cdsGeoZones.Open;
cdsGeoZones.First;

Not all the datasets need to be managed for the whole application duration. Only those that need are managed by ClientDataSets. The "First" calls happen because I don't know if the server back end will change. Some database drivers, DLLs, (expecially) ODBC connectors etc. etc. don't perform the actual heavy lifting during Open but at the first cursor operation. Therefore I make sure it happens, even if the current driver does not strictly need it. The Sleep(100) is there to let the users and the developers see the messages when opening small tables. May be removed once the software is final, of course.

The Lock, try / finally clauses etc. are there to remind you that we are in a thread and some resources are best accessed with some precautions. In this specific case we have other threads (irrelevant for this article, thus not covered) so we have to protect some data structures. In particular, I have "borrowed" the basic Delphi thread safe list locking mechanism paradygm so the method names are also the same.

OmniThreadLibrary based preload module thread worker

Here is the most relevant / didactic code:

type
  TDBPreloadWorker = class(TOmniWorker)
  protected
    ThreadModDatabase : TModDBPreload;
    FStatusString : string;
  public
    constructor Create;
    function Initialize : boolean; override;
    procedure Cleanup; override;
    procedure SendStatusMessage(anID : Word; aValue : string = ''); overload;
    procedure SendStatusMessage(aValue : string); overload;
    procedure DisconnectFromDatabase;

    procedure OMSendMessage(var msg: TOmniMessage); message MSG_SEND_MESSAGE;
    procedure OMDisconnectFromDatabase(var msg: TOmniMessage); message MSG_DISCONNECT_FROM_DATABASE;
    procedure OMUpdateStateMachine(var msg: TOmniMessage); message MSG_UPDATE_STATE_MACHINE;
  end;
...

constructor TDBPreloadWorker.Create;
begin
  Inherited;
  FStatusString := 'Connecting to server...';
  ThreadModDatabase := Nil;
end;

function TDBPreloadWorker.Initialize : boolean;
begin
  ThreadModDatabase := TModDBPreload.Create(Nil);
  ModDBPreload := ThreadModDatabase;
  ThreadModDatabase.DBPreloadWorker := Self;
  DisconnectFromDatabase; // In case of leftover Active := true from designing the software
  Result := true;
end;

procedure TDBPreloadWorker.Cleanup;
begin
  DisconnectFromDatabase;
  ThreadModDatabase.Free;
  ThreadModDatabase := Nil;
end;

procedure TDBPreloadWorker.SendStatusMessage(anID : Word; aValue : string);
begin
  FStatusString := aValue; // Stored in case the main application polls a status update
  Task.Comm.Send(anID, aValue);
end;

procedure TDBPreloadWorker.SendStatusMessage(aValue : string);
begin
  SendStatusMessage(MSG_GENERAL_RESPONSE, aValue);
end;

procedure TDBPreloadWorker.DisconnectFromDatabase;
begin
  if Assigned(ThreadModDatabase) then
    ThreadModDatabase.DisconnectFromDatabase;
end;

procedure TDBPreloadWorker.OMSendMessage(var msg: TOmniMessage);
begin
  Task.Comm.Send(MSG_GENERAL_RESPONSE, FStatusString);
end;

procedure TDBPreloadWorker.OMDisconnectFromDatabase(var msg: TOmniMessage);
begin
  ...
  DisconnectFromDatabase;
end;

procedure TDBPreloadWorker.OMSendMessage(var msg: TOmniMessage);
begin
  Task.Comm.Send(MSG_GENERAL_RESPONSE, FStatusString);
end;

procedure TDBPreloadWorker.OMUpdateStateMachine(var msg: TOmniMessage);
begin
  Task.Comm.Send(MSG_GENERAL_RESPONSE, FStatusString); // Needed to show the pre-loaded status

  if Assigned(ThreadModDatabase) then
  begin
    try
      ThreadModDatabase.ConnectToDatabase;
      SendStatusMessage('Reading database tables...');

      if not ThreadModDatabase.QueryExecute then
      begin
        raise Exception.Create('Consistency check: the database does not return the expected values');
      end;

      SendStatusMessage(MSG_SUCCESS, 'Tables have been succesfully read');
      SendStatusMessage(MSG_TASK_COMPLETED);

    except
      On E : Exception do
      begin
        DisconnectFromDatabase;
        SendStatusMessage(MSG_TASK_FAILURE, E.Message);
      end;
    end;
  end;
end;

Some code deserves further explanations:

function TDBPreloadWorker.Initialize : boolean;

creates the preload data module. That is, everything is self contained in the thread's context and does not clash with others.

procedure TDBPreloadWorker.SendStatusMessage(anID : Word; aValue : string);

this is how to send a message (by the way, it's not limited to strings) to the main VCL thread by means of the OmniThreadLibrary.

procedure TDBPreloadWorker.OMUpdateStateMachine(var msg: TOmniMessage);

this is the main preload data module initialization management code. It performs handshaking with the VCL main thread and basically plays as one of the state machines I have implemented in the program.

For those wondering where all those constants come from: they are declared in a separate file included by all the threads related classes. They are simple, free to choose integers:

const
  MSG_GENERAL_RESPONSE         = 0;
  MSG_SEND_MESSAGE             = 1;
  MSG_SHUTDOWN                 = 2;
  MSG_SUCCESS                  = $20;
  MSG_ABORT                    = $30;
  MSG_RETRY                    = $31;
  MSG_TASK_COMPLETED           = $40;
  MSG_FAILURE                  = $8020;
  MSG_ABORTED                  = $8030;
  MSG_TASK_FAILURE             = $8040;
  MSG_UPDATE_STATE_MACHINE     = 9;
  MSG_TIMER_1                  = 10;
  MSG_DISCONNECT_FROM_DATABASE = 99;

Main form side preload management code

The various threads are spawned at program start. A TOmniEventMonitor has its OnTaskMessage event pointing to:

procedure TFrmMain.monDBPreloadTaskMessage(const task: IOmniTaskControl;
  const msg: TOmniMessage);
var
  MessageString : string;
  ComponentsNewState : boolean;

begin
  MessageString := msg.MsgData.AsString;

  if Length(MessageString) > 0 then
    UpdateStatusBar(MessageString);

  if task = FDBPreloadWorkerControl then
  begin
    if (msg.MsgID = MSG_TASK_COMPLETED) or (msg.MsgID = MSG_TASK_FAILURE) then
    begin
      ComponentsNewState := (msg.MsgID = MSG_TASK_COMPLETED);

      // Unlike for the watchdog, the preload thread is not terminated
      // The data is needed by the program till its end
      // DBPreloadTerminate;

      // Lets the main database queries be started
      DBPreloadSuccess := (msg.MsgID = MSG_TASK_COMPLETED);
      MainViewEnabled := ComponentsNewState;

      if msg.MsgID = MSG_TASK_FAILURE then
      begin
        if MessageDlg('Unable to load the data tables from the database server', mtError, [mbRetry, mbAbort], 0) = mrAbort then
          Close
        else
          // Reinitialize the preload thread.
          ...
      end;
    end;
  end;
end;

This is the totally simple procedure that in the end gets called in order to update the main form's status bar:

procedure TFrmMain.UpdateStatusBar(Value : string);
begin
  pnlStatusBar.SimpleText := Value;
  pnlStatusBar.Update;
  Application.ProcessMessages;
end;

Main database module management code

Last but not least, here is how to actually "attach" to the preload data module ClientDataSets. Call this code from the main form and the foundations of your application are basically done!

procedure TModDatabase.ConnectToDatabase;

  procedure ConnectDataSet(CDS : TClientDataSet; PreloadDataSet : TClientDataSet; RuntimeDataSet : TZAbstractRODataset; SetLanguage : boolean = false);
  begin
   // Only required by datasets needing a locale_id parameter
    if (SetLanguage) then
    begin
      CDS.Params.ParamByName('language_id').AsInteger := ModApplicationCommon.ApplicationLocaleInfo.LocaleIDForQueries;
      RuntimeDataSet.ParamByName('language_id').AsInteger := ModApplicationCommon.ApplicationLocaleInfo.LocaleIDForQueries;
    end;

    CDS.Data := PreloadDataSet.Data;
    CDS.Active := true;
  end;

begin
  DisconnectFromDatabase;
  dbcEShop.Connect;

  UpdateStatusBar('Setting up products archive');
  ConnectDataSet(cdsProduct, ModDBPreload.cdsProduct, qryProduct, true);
  UpdateStatusBar('Setting up products options archive');
  ConnectDataSet(cdsProductOption, ModDBPreload.cdsProductOption, qryProductOption);
  UpdateStatusBar('Setting up options archive');
  ConnectDataSet(cdsOption, ModDBPreload.cdsOption, qryOption);
  UpdateStatusBar('Setting up options descriptions archive');
  ConnectDataSet(cdsOptionDescription, ModDBPreload.cdsOptionDescription, qryOptionDescription, true);
  ...

I hope to have posted enough information to give out an idea about the whole process. Please feel free to ask any questions and sorry for the lexicon, English is my fourth language.