The best practice on how to wait for data within thread and write it into file?

Question

I am programming multi-threaded app. I have two threads. On is for transferring some data from device to global data buffer and second is for writing those data to file.

Data from device to buffer is transferring asynchronously. The purpose of second thread should be to wait for specified amount of data to be written in main data buffer and finally to write it to file.

Well the first thread is in DLL and second one is in main app. Temporarily I solve this with events. First thread transfers data from device to main data buffer and counts data and when specified amount of data is transferred it sets an event. The second one waits for event to be signalled and when it is it runs some code for data store. Simple as that it is working.

  Thread1.Execute:

  var DataCount, TransferedData: Integer;

  DataCounter := 0;
  while not Terminted do
  begin
    TransferData(@pData, TransferedData);
    Inc(DataCounter, TransferedData)
    if DataCounter >= DataCountToNotify then SetEvent(hDataCount);
  end;



  Thread2.Execute:

  hndlArr[0] := hDataCount;
  hndlArr[1] := hTerminateEvent;

  while (not Terminated) do
  begin
    wRes := WaitForMultipleObjects(HandlesCount, Addr(hndlArr), false, 60000);
    case wRes of
      WAIT_OBJECT_0:
        begin
          Synchronize(WriteBuffer);                   // call it from main thread
          ResetEvent(hndlArr[0]);
        end;
      WAIT_OBJECT_0 + 1:
        begin
          ResetEvent(hTerminateEvent);
          break;
        end;
      WAIT_TIMEOUT:  Break;
    end;
  end;

Now I would like to make second thread more independent... so I can make multiple instances of second thread and I don't need to wait for first thread. I would like to move data counting part of code from first thread to second one so I won't need data counting in first thread anymore. First one will be just for data transfer purpose.

I would like to use second one as data counter and to store data. But now I will have to loop and constantly check manually for specified data amount. If I had while loop I will have to add some sleep so second thread will not decrease computer performances but I don't know how long should sleep be while speed of data transfer in firts thread is not constant and thus speed of counting in second thread will vary.

My guess that this code sample is not good:

  Thread2.Execute:
  var DataCount: Integer;
  DataIdx1 := GetCurrentDataIdx; 
  while (not Terminated) do
  begin     
    if (GetCurrentDataIdx - DataIdx1) >= DataCountToNotify then 
    begin
      Synchronize(WriteBuffer);
      DataIdx1 := GetCurrentIdx;
    end;
    sleep(???);
  end;

So my question is what is the best approach to solve that issue with data counting and storing it within second thread? What are your experiences and suggestions?

Answer 1

You have some issues. @LU RD has already pointed one out - don't synchronize stuff that does not need to be synchronized. It's not clear what 'WriteBuffer' does, but the file system and all databases I have used are just fine to have one thread opening a file/table and writing to them.

Your buffer system could probably do with some attention. Is there some 'specified data amount' or is this some notional figure that allows lazy writing?

Typically, producer and consumer threads exchange multiple buffer pointers on queues and so avoid sharing any single buffer. Given that this is a DLL and so memory-management calls can be problematic, I would probably avoid them as well by creating a pool of buffers at startup to transfer data round the system. I would use a buffer class rather than just pointers to memory, but it's not absolutely required, (just much more easy/flexible/safe).

Sleep() loops are a spectacularly bad way of communicating between threads. Sleep() does hae its uses, but this is not one of them. Delphi/Windows has plenty of synchronization mechanisms - events, semaphores, mutexes etc - tha make such polling unnecessary.

LU RD has also mentioned the problems of parallel processing of data whose order must be preserved. This often requires yet another thread, a list-style collection and sequence-numbers. I wouldn't try that until you have the inter-thread comms working well.