Limit number of Threads in Task Parallel Library

Question

I have few hundreds of files i need to upload to Azure Blob Storage.
I want to use parallel task library.
But instead of running all the 100 threads to upload in a foreach on list of files, how can i put a limit on max number of threads that it can use and finish the job in parallel. or does it balance the things automatically?

Answer 1

You should not be using threads for this at all. There's a Task-based API for this, which is naturally asynchronous: CloudBlockBlob.UploadFromFileAsync. Use it with async/await and SemaphoreSlim to throttle the number of parallel uploads.

Example (untested):

const MAX_PARALLEL_UPLOADS = 5;

async Task UploadFiles()
{
    var files = new List<string>();
    // ... add files to the list

    // init the blob block and
    // upload files asynchronously
    using (var blobBlock = new CloudBlockBlob(url, credentials))
    using (var semaphore = new SemaphoreSlim(MAX_PARALLEL_UPLOADS))
    {
        var tasks = files.Select(async(filename) => 
        {
            await semaphore.WaitAsync();
            try
            {
                await blobBlock.UploadFromFileAsync(filename, FileMode.Create);
            }
            finally
            {
                semaphore.Release();
            }
        }).ToArray();

        await Task.WhenAll(tasks);
    }
}

Answer 2

Did you try use MaxDegreeOfParallelism? Like this:

System.Threading.Tasks.Parallel.Invoke(
new Tasks.ParallelOptions {MaxDegreeOfParallelism =  5 }, actionsArray)