How to limit I/O operations in .NET application?

Question

I'm developing an application (.NET 4.0, C#) that:
1. Scans file system.
2. Opens and reads some files.

The app will work in background and should have low impact on the disk usage. It shouldn't bother users if they are doing their usual tasks and the disk usage is high. And vice versa, the app can go faster if nobody is using the disk.
The main issue is I don't know real amount and size of I/O operations because of using API (mapi32.dll) to read files. If I ask API to do something I don't know how many bytes it reads to handle my response.

So the question is how to monitor and manage the disk usage? Including file system scanning and files reading...

Check performance counters that are used by standard Performance Monitor tool? Or any other ways?

Answer 1

Using the System.Diagnostics.PerformanceCounter class, attach to the PhysicalDisk counter related to the drive that you are indexing.

Below is some code to illustrate, although its currently hard coded to the "C:" drive. You will want to change "C:" to whichever drive your process is scanning. (This is rough sample code only to illustrate the existence of performance counters - don't take it as providing accurate information - should always be used as a guide only. Change for your own purpose)

Observe the % Idle Time counter which indicates how often the drive is doing anything. 0% idle means the disk is busy, but does not necessarily mean that it is flat-out and cannot transfer more data.

Combine the % Idle Time with Current Disk Queue Length and this will tell you if the drive is getting so busy that it cannot service all the requests for data. As a general guideline, anything over 0 means the drive is probably flat-out busy and anything over 2 means the drive is completely saturated. These rules apply to both SSD and HDD fairly well.

Also, any value that you read is an instantaneous value at a point in time. You should do a running average over a few results, e.g. take a reading every 100ms and average 5 readings before using the information from the result to make a decision (i.e., waiting until the counters settle before making your next IO request).

internal DiskUsageMonitor(string driveName)
{

    // Get a list of the counters and look for "C:"

    var perfCategory = new PerformanceCounterCategory("PhysicalDisk");
    string[] instanceNames = perfCategory.GetInstanceNames();

    foreach (string name in instanceNames)
    {
        if (name.IndexOf("C:") > 0)
        {
            if (string.IsNullOrEmpty(driveName))
               driveName = name;
        }
    }


    _readBytesCounter = new PerformanceCounter("PhysicalDisk", 
                                               "Disk Read Bytes/sec", 
                                               driveName);

    _writeBytesCounter = new PerformanceCounter("PhysicalDisk", 
                                                "Disk Write Bytes/sec", 
                                                driveName);

    _diskQueueCounter = new PerformanceCounter("PhysicalDisk", 
                                               "Current Disk Queue Length", 
                                               driveName);

    _idleCounter = new PerformanceCounter("PhysicalDisk",
                                          "% Idle Time", 
                                          driveName);
    InitTimer();
}

internal event DiskUsageResultHander DiskUsageResult;

private void InitTimer()
{
    StopTimer();
    _perfTimer = new Timer(_updateResolutionMillisecs);
    _perfTimer.Elapsed += PerfTimerElapsed;
    _perfTimer.Start();
}

private void PerfTimerElapsed(object sender, ElapsedEventArgs e)
{
    float diskReads = _readBytesCounter.NextValue();
    float diskWrites = _writeBytesCounter.NextValue();
    float diskQueue = _diskQueueCounter.NextValue();
    float idlePercent = _idleCounter.NextValue();

    if (idlePercent > 100)
    {
        idlePercent = 100;
    }

    if (DiskUsageResult != null)
    {
        var stats = new DiskUsageStats
                        {
                                DriveName = _readBytesCounter.InstanceName,
                                DiskQueueLength = (int)diskQueue,
                                ReadBytesPerSec = (int)diskReads,
                                WriteBytesPerSec = (int)diskWrites,
                                DiskUsagePercent = 100 - (int)idlePercent
                        };
        DiskUsageResult(stats);
    }
}

Answer 2

A long term ago Microsoft Research published a paper on this (sorry I can’t remember the url).
From what I recall:

• The program started off doing very few "work items".
• They measured how long it took for each of their "work item".
• After running for a bit, they could work out how fast an "work item" was with no load on the system.
• From then on, if the "work item" were fast (e.g. no other programmers making requests), they made more requests, otherwise they backed-off

The basic ideal is:

“if they are slowing me down, then I must be slowing them down, so do less work if I am being slowed down”

Answer 3

Something to ponder: what if there are other processes which follow the same (or a similar) strategy? Which one would run during the "idle time"? Would the other processes get a chance to make use of the idle time at all?

Obviously this can't be done correctly unless there is some well-known OS mechanism for fairly dividing resources during idle time. In windows, this is done by calling SetPriorityClass.

This document about I/O prioritization in Vista seems to imply that IDLE_PRIORITY_CLASS will not really lower the priority of I/O requests (though it will reduce the scheduling priority for the process). Vista added new PROCESS_MODE_BACKGROUND_BEGIN and PROCESS_MODE_BACKGROUND_END values for that.

In C#, you can normally set the process priority with the Process.PriorityClass property. The new values for Vista are not available though, so you'll have to call the Windows API function directly. You can do that like this:

[DllImport("kernel32.dll", CharSet=CharSet.Auto, SetLastError=true)]
public static extern bool SetPriorityClass(IntPtr handle, uint priorityClass);

const uint PROCESS_MODE_BACKGROUND_BEGIN = 0x00100000;

static void SetBackgroundMode()
{
   if (!SetPriorityClass(new IntPtr(-1), PROCESS_MODE_BACKGROUND_BEGIN))
   {
      // handle error...
   }
}

I did not test the code above. Don't forget that it can only work on Vista or better. You'll have to use Environment.OSVersion to check for earlier operating systems and implement a fall-back strategy.