My program has a list of 200k files. I have to import each to the database. I takes a long time so I started researching about multithreads as a means to speed up the importing process. I finally got to an implementation but I'm not sure it's actually working.
After using Workaround for the WaitHandle.WaitAll 64 handle limit? as a sample for my c# code I've came up with:
int threadCount = 0;
for (int i = 0; i < this.Total; i++)
{
Finished = new ManualResetEvent(false);
threadCount = this.ThreadCount;
Interlocked.Increment(ref threadCount);
FileHandler fh = new FileHandler(finished, sorted[i], this.PicturesFeatures, this.Outcome, this.SiteIds, this.LastId, this.Order, this.ThreadCount);
Console.Write(i + " ");
ThreadPool.QueueUserWorkItem(new WaitCallback(HandleFile), fh);
Console.Write(i + " ");
Finished.WaitOne();
}
And HandleFile() goes as:
private void HandleFile(object s)
{
try
{
//code
}
finally
{
if (Interlocked.Decrement(ref threadCount) == 0)
{
Finished.Set();
}
}
}
I've put those console.Write thinking that if a process is longer it would finish later than some other ("0 0 1 2 2 1 3 3 ..."), but it's always in order ("0 0 1 1 2 2 3 3 4 4 ...")
Concurrency and Parallelism Concurrency indicates that more than one thread is making progress, but the threads are not actually running simultaneously. The switching between threads happens quickly enough that the threads might appear to run simultaneously.
Within a process or program, we can run multiple threads concurrently to improve the performance. Threads, unlike heavyweight process, are lightweight and run inside a single process – they share the same address space, the resources allocated and the environment of that process.
Multithreading : The ability of a central processing unit (CPU) (or a single core in a multi-core processor) to provide multiple threads of execution concurrently, supported by the operating system [3]. Multiprocessing: The use of two or more CPUs within a single computer system [4][5].
Your output is to be expected. You're writing the output in the main thread, the QueueUserWorkItem
function does not block, it registers your HandleFile
function to be executed in a separate thread. So regardless of how long the work items take, your prints will happen in the expected order as they are all from the main thread.
Additionally, you're not getting the benefit of parallelism with this code because you're waiting after every item you submit. You're essentially saying I won't submit my next work item until the last one is finished. This is just a more complicated way of writing normal serialized code. In order to introduce parallelism, you need to add multiple items to the queue without waiting in between.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With