I need an contributon function for my win service instances

Question

I have an table with Guid as primary key. The table has contained a lot of rows already. Also, I have an win service, that do some set of actions with each row (possible needs read and write data from another databases). So processing of one row takes a quite lot of time. (in average about 100 seconds)

My win service works like this:

public class MyDto
{
     public Guid Id { get; set; }
}

while (true){
     if(time to start){
          List<MyDto> rows = LoadData();
          foreach(MyDto obj in rows){
               Process(obj);//it takes in average about 100 sec
          }
     }
}

I need reduce performing time of all my rows. By some reasons I decided increase insatnces of my win service. So I need that each win service will proccess own row set.

I've parametrized my LoadData() fun:

public List<MyDto> LoadData(int winServInstanceNumber){

}

So i need an contribution function depends from total win service instances count and concreate win service instance number.

Can you offer something better than

//on .net side
obj.Id.GetHashCode()%totalWinServiceInstancesCount

or

--on sql side
HASHBYTES('MD5', CAST(id as varbinary(16)))%totalWinServiceInstancesCount

Answer 1

Looks like all you need is to spin more threads to process your data. But to do this, you need a control of what you processing, in order not to process same thing twice. To get a control you can use MSMQ, for example, or System.Collections.Queue. Your service should be responsible to query database and load unprocessed rows into your queue.

Then, you can call some static method ProcessBatch. It will go to the queue and spin a thread(s), and pass the id(s) of the row(s) to the processor(s)/Workers. worker will only process one row. Worker can be separate EXE and run out of process. Your 'ProcessBatch' should control, what processed/not processed. It should control how many threads are currently running. You don't want to spin too many.

So

Service           ProcessControl          Worker
  |                       |                |
  |---Load Queue          |                |
  |         |             |                |
  |<--------|             |                |
  |                       |                |
  |-----Call When Q ----->|---Queue        |
  |                       |       |        |
  |                       |<------|        |
  |                       |                |
  |---Load Queue          |----Start------>|
  |        |              |<---Success-----|
  |<-------|              |                |
  |                       |---Permanent    |
  |-----Call When Q ----->|    Dequeue     |
  |                       |       |        |
  |                       |<------|        |

This is probably typical workload split that speeds up otherwise slow processes