Unable to read final few Kb of logical drives on Windows 7 64-bit

Question

I'm having some problems addressing logical drives. For clarity, my definition of 'Physical Disk' (PD) is the raw disk regardless of partitioning. 'Logical Drive' (LD) refers to a volume such as Drive E:, Drive F: etc.

Using the examples from RRUZ (my hero SO member) and implementing the WMI Class I have created a Freepascal program for reading disks. I address PD by \.\PhyscialDiskX and that works fine by the examples created by RRUZ (here). I can read all the bytes no problem for PD's.

I use the same handle technique for logical volumes, which are \?\E: or \?\F: etc. I then use IOCTL_DISK_GET_LENGTH_INFO to get the length of the PD or LV and then read the byte range until ReadBytes = TotalLength. I read on the MSDN website that it will automatically retrieve the size of whatever device handle it is passed - PD or LD alike. And indeed I have checked the szie values returned by my program againzt WinHex, FTK Imager, HxD and several other low level disk tools. With the exception of 1 byte variances caused by zero or 1 starting positions, they match.

However, for some reason, my program is failing to acquire the final 32Kb on Windows 7 Pro 64-bit, despite running the program as administrator. It reads the whole disk and then on the final buffer read (which is done as 64Kb buffers) BytesRead returns -1. Using the debugger I worked out the following values :

493,846,527 exact LV size of Drive F:
493,813,760 total bytes read at time of failure
32,767 bytes missing

The result of the following

BytesRead    := FileRead(hDiskHandle, Buffer, (DiskSize - TotalBytesRead));

is -1 on the final buffer read. THis the line that tests for the end of the disk by saying "if the amount left to read is less than the size of the buffer size, only try to read what is left". So, the value of bytes being asked to be stored by FileRead at the end is 32,767 (because DiskSize - TotalBytesRead at that point is 32,767, meaning that's how many bytes are left to read of the disk). The designated size of buffer is 64Kb. My understanding is that you can put less in a buffer than it is capable of holding but not more (FileRead states : "Buffer must be at least Count bytes long. No checking on this is performed"? IS that correct? If it is not then this may be (and probably is) the issue.

I don't know if it's due to IOCTL_DISK_GET_LENGTH_INFO, the buffer storage or something else? Hoping someone can help? I have also posted along with some screenshot at the Lazarus Freepascal forum. Here is my relevant code sections:

The handle:

// Create handle to source disk. Abort if fails
  hSelectedDisk := CreateFileW(PWideChar(SourceDevice), FILE_READ_DATA,
                   FILE_SHARE_READ, nil, OPEN_EXISTING, FILE_FLAG_SEQUENTIAL_SCAN, 0);

  if hSelectedDisk = INVALID_HANDLE_VALUE then
  begin
    RaiseLastOSError;
  end 

Compute the size ion bytes of the given device:

ExactDiskSize := GetDiskLengthInBytes(hSelectedDisk);

Now read the device and store the input as a flat file

ImageResult   := WindowsImageDisk(hSelectedDisk, ExactDiskSize, HashChoice, hImageName);

Functions for the above:

function GetDiskLengthInBytes(hSelectedDisk : THandle) : Int64;
const
  // These are defined at the MSDN.Microsoft.com website for DeviceIOControl
  // and https://forum.tuts4you.com/topic/22361-deviceiocontrol-ioctl-codes/
  {
  IOCTL_DISK_GET_DRIVE_GEOMETRY      = $0070000
  IOCTL_DISK_GET_PARTITION_INFO      = $0074004
  IOCTL_DISK_SET_PARTITION_INFO      = $007C008
  IOCTL_DISK_GET_DRIVE_LAYOUT        = $007400C
  IOCTL_DISK_SET_DRIVE_LAYOUT        = $007C010
  IOCTL_DISK_VERIFY                  = $0070014
  IOCTL_DISK_FORMAT_TRACKS           = $007C018
  IOCTL_DISK_REASSIGN_BLOCKS         = $007C01C
  IOCTL_DISK_PERFORMANCE             = $0070020
  IOCTL_DISK_IS_WRITABLE             = $0070024
  IOCTL_DISK_LOGGING                 = $0070028
  IOCTL_DISK_FORMAT_TRACKS_EX        = $007C02C
  IOCTL_DISK_HISTOGRAM_STRUCTURE     = $0070030
  IOCTL_DISK_HISTOGRAM_DATA          = $0070034
  IOCTL_DISK_HISTOGRAM_RESET         = $0070038
  IOCTL_DISK_REQUEST_STRUCTURE       = $007003C
  IOCTL_DISK_REQUEST_DATA            = $0070040
  IOCTL_DISK_CONTROLLER_NUMBER       = $0070044
  IOCTL_DISK_GET_PARTITION_INFO_EX   = $0070048
  IOCTL_DISK_SET_PARTITION_INFO_EX   = $007C04C
  IOCTL_DISK_GET_DRIVE_LAYOUT_EX     = $0070050
  IOCTL_DISK_SET_DRIVE_LAYOUT_EX     = $007C054
  IOCTL_DISK_CREATE_DISK             = $007C058
  IOCTL_DISK_GET_LENGTH_INFO         = $007405C  // Our constant...
  SMART_GET_VERSION                  = $0074080
  SMART_SEND_DRIVE_COMMAND           = $007C084
  SMART_RCV_DRIVE_DATA               = $007C088
  IOCTL_DISK_GET_DRIVE_GEOMETRY_EX   = $00700A0
  IOCTL_DISK_UPDATE_DRIVE_SIZE       = $007C0C8
  IOCTL_DISK_GROW_PARTITION          = $007C0D0
  IOCTL_DISK_GET_CACHE_INFORMATION   = $00740D4
  IOCTL_DISK_SET_CACHE_INFORMATION   = $007C0D8
  IOCTL_DISK_GET_WRITE_CACHE_STATE   = $00740DC
  IOCTL_DISK_DELETE_DRIVE_LAYOUT     = $007C100
  IOCTL_DISK_UPDATE_PROPERTIES       = $0070140
  IOCTL_DISK_FORMAT_DRIVE            = $007C3CC
  IOCTL_DISK_SENSE_DEVICE            = $00703E0
  IOCTL_DISK_INTERNAL_SET_VERIFY     = $0070403
  IOCTL_DISK_INTERNAL_CLEAR_VERIFY   = $0070407
  IOCTL_DISK_INTERNAL_SET_NOTIFY     = $0070408
  IOCTL_DISK_CHECK_VERIFY            = $0074800
  IOCTL_DISK_MEDIA_REMOVAL           = $0074804
  IOCTL_DISK_EJECT_MEDIA             = $0074808
  IOCTL_DISK_LOAD_MEDIA              = $007480C
  IOCTL_DISK_RESERVE                 = $0074810
  IOCTL_DISK_RELEASE                 = $0074814
  IOCTL_DISK_FIND_NEW_DEVICES        = $0074818
  IOCTL_DISK_GET_MEDIA_TYPES         = $0070C00
  }
  IOCTL_DISK_GET_LENGTH_INFO  = $0007405C;
type
  TDiskLength = packed record
    Length : Int64;
  end;

var
  BytesReturned: DWORD;
  DLength: TDiskLength;
  ByteSize: int64;

begin
  BytesReturned := 0;
  // Get the length, in bytes, of the physical disk
  if not DeviceIOControl(hSelectedDisk,  IOCTL_DISK_GET_LENGTH_INFO, nil, 0,
         @DLength, SizeOf(TDiskLength), BytesReturned, nil) then
           raise Exception.Create('Unable to determine byte capacity of disk.');
  ByteSize := DLength.Length;
  ShowMessage(IntToStr(ByteSize));
  result := ByteSize;
end;

The disk reader function

function WindowsImageDisk(hDiskHandle : THandle; DiskSize : Int64; HashChoice : Integer; hImageName : THandle) : Int64;
var
  Buffer                   : array [0..65535] of Byte;   // 1048576 (1Mb) or 262144 (240Kb) or 131072 (120Kb buffer) or 65536 (64Kb buffer)

  BytesRead                : integer;

  NewPos, SectorCount,
  TotalBytesRead, BytesWritten, TotalBytesWritten : Int64;
  ... 
 // Now to seek to start of device
      FileSeek(hDiskHandle, 0, 0);
        repeat
          // Read device in buffered segments. Hash the disk and image portions as we go
          if (DiskSize - TotalBytesRead) < SizeOf(Buffer) then
            begin
              // Read 65535 or less bytes
              BytesRead    := FileRead(hDiskHandle, Buffer, (DiskSize - TotalBytesRead));
              BytesWritten := FileWrite(hImageName, Buffer, BytesRead);
            end
          else
            begin
              // Read 65536 (64kb) at a time
              BytesRead     := FileRead(hDiskHandle, Buffer, SizeOf(Buffer));
              BytesWritten  := FileWrite(hImageName, Buffer, BytesRead);
            end;
          if BytesRead = -1 then
            begin
              ShowMessage('There was a read error encountered. Aborting');
              // ERROR IS THROWN AT THIS POINT ONLY WITH LD's - not PD's
              exit;
            end
          else
          begin
          inc(TotalBytesRead, BytesRead);
          inc(TotalBytesWritten, BytesWritten);
          NewPos := NewPos + BytesRead;   
...
  until (TotalBytesRead = DiskSize);

Answer 1

Probably, it is a boundary check error. Quote from MSDN (CreateFile, note on opening physical drives and volumes, which you call logical drives):

To read or write to the last few sectors of the volume, you must call DeviceIoControl and specify FSCTL_ALLOW_EXTENDED_DASD_IO

Answer 2

I suspect that the problem stems from the use of 64-bit integers and arithmetic to calculate a value passed as a 32-bit Integer:

FileRead(hDiskHandle, Buffer, (DiskSize - TotalBytesRead));

I cannot explain with certainty why this might affect only LD's and not PD's, except to speculate that there may be some difference in the reported DiskSize that somehow avoids the Int64 arithmetic/32-bit problem in that case.

e.g. If the 32-bit truncated result of the Int64 arithmetic is NEGATIVE (which requires only that the high bit be set, i.e. 1 not 0) then FileRead() will return -1 since a negative value for "bytes to read" is invalid.

But if the high-bit in the result is NOT set, resulting in a positive value, then even if that value is significantly greater than 64KB this will not cause an error since this invocation is called only when you have already determined that there are fewer than 64K bytes to be read. The 32-bit truncated Int64 arithmetic may result in a request to read 2 BILLION bytes but FileRead() is only going to read the actual 32K bytes that remain anyway.

However, this very fact points to a solution (assuming that this diagnosis is correct).

As noted, FileRead() (which is just a wrapper around ReadFile(), on Windows) will read either the number of bytes specified or as many bytes remain to be read, whichever is lower.

So if you specify 64KB but only 32KB remain, then only 32KB will be read.

You can replace all of this code:

if (DiskSize - TotalBytesRead) < SizeOf(Buffer) then
begin
  // Read 65535 or less bytes
  BytesRead    := FileRead(hDiskHandle, Buffer, (DiskSize - TotalBytesRead));
  BytesWritten := FileWrite(hImageName, Buffer, BytesRead);
end
else
begin
  // Read 65536 (64kb) at a time
  BytesRead     := FileRead(hDiskHandle, Buffer, SizeOf(Buffer));
  BytesWritten  := FileWrite(hImageName, Buffer, BytesRead);
end;

With simply:

BytesRead     := FileRead(hDiskHandle, Buffer, SizeOf(Buffer));
BytesWritten  := FileWrite(hImageName, Buffer, BytesRead);

This eliminates the 64-bit arithmetic and any possibility for errors resulting from 64-bit results being passed in 32-bit values. If the final segment contains only 32KB, then only 32KB will be read.

You can also simplify your loop termination (removing the 64-bit arithmetic, unless you need the accumulated values for other purposes). Instead of accumulating the total bytes read, you can terminate your loop simply when your FileRead() reads less than the number of bytes specified. i.e. BytesRead < 64KB:

Even if your disk is an exact multiple of 64KB blocks, your penultimate FileRead() will return a full buffer of 64KB, and the very next FileRead() will read 0 bytes, which is < 64KB, terminating the loop. :)