Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Limit the number of parallel threads in C#

Tags:

I am writing a C# program to generate and upload a half million files via FTP. I want to process 4 files in parallel since the machine have 4 cores and the file generating takes much longer time. Is it possible to convert the following Powershell example to C#? Or is there any better framework such as Actor framework in C# (like F# MailboxProcessor)?

Powershell example

$maxConcurrentJobs = 3;

# Read the input and queue it up
$jobInput = get-content .\input.txt
$queue = [System.Collections.Queue]::Synchronized( (New-Object System.Collections.Queue) )
foreach($item in $jobInput)
{
    $queue.Enqueue($item)
}

# Function that pops input off the queue and starts a job with it
function RunJobFromQueue
{
    if( $queue.Count -gt 0)
    {
        $j = Start-Job -ScriptBlock {param($x); Get-WinEvent -LogName $x} -ArgumentList $queue.Dequeue()
        Register-ObjectEvent -InputObject $j -EventName StateChanged -Action { RunJobFromQueue; Unregister-Event $eventsubscriber.SourceIdentifier; Remove-Job $eventsubscriber.SourceIdentifier } | Out-Null
    }
}

# Start up to the max number of concurrent jobs
# Each job will take care of running the rest
for( $i = 0; $i -lt $maxConcurrentJobs; $i++ )
{
    RunJobFromQueue
}

Update:
The connection to remote FTP server can be slow so I want to limit the FTP uploading processing.

like image 372
ca9163d9 Avatar asked Jan 13 '12 16:01

ca9163d9


People also ask

How do I limit the number of threads in C#?

To set the lower limit of the threads in the thread pool you can use the ThreadPool. SetMinThreads property. The default lower limit of the number of threads in the thread pool is one thread per processor.

How many threads will parallel ForEach use?

Example using Degree of Parallelism in C# to Restrict the number of Threads. In the below example, we have set MaxDegreeOfParallelism to 2 which means a maximum of 2 threads are going to execute our parallel foreach loop.

Do C threads run in parallel?

Yes, if you have multiple processors or multi-core processors. One thread will run in one core.

Can multiple threads run in parallel?

When multi-threaded program execution occurs on a multiple core system (multiple uP, or multiple multi-core uP) threads can run concurrently, or in parallel as different threads may be split off to separate cores to share the workload. This is one example of parallel processing.


3 Answers

Assuming you're building this with the TPL, you can set the ParallelOptions.MaxDegreesOfParallelism to whatever you want it to be.

Parallel.For for a code example.

like image 149
Austin Salonen Avatar answered Dec 12 '22 16:12

Austin Salonen


Task Parallel Library is your friend here. See this link which describes what's available to you. Basically framework 4 comes with it which optimises these essentially background thread pooled threads to the number of processors on the running machine.

Perhaps something along the lines of:

ParallelOptions options = new ParallelOptions();

options.MaxDegreeOfParallelism = 4;

Then in your loop something like:

Parallel.Invoke(options,
 () => new WebClient().Upload("http://www.linqpad.net", "lp.html"),
 () => new WebClient().Upload("http://www.jaoo.dk", "jaoo.html"));
like image 45
Jeb Avatar answered Dec 12 '22 16:12

Jeb


If you are using .Net 4.0 you can use the Parallel library

Supposing you're iterating throug the half million of files you can "parallel" the iteration using a Parallel Foreach for instance or you can have a look to PLinq Here a comparison between the two

like image 26
Giorgio Minardi Avatar answered Dec 12 '22 16:12

Giorgio Minardi