Determine the number of "logical" bytes read/written in a Linux system

Question

I would like to determine the number of bytes logically read/written by all processes via syscalls such as read() and write(). This is different than the number of bytes actually fetched from the storage layer (displayed by tools like iotop) since it includes (for example) reads that hit the pagecache, and is also differs in when writes are recognized: the logical write IO happens immediately when the write call is issued, while the actual physical IO may occur some time later depending on various factors (Linux usually buffers writes and does the physical IO some time later).

I know how to do it on a per-process basis (see this question for example), but not how to the get the system-wide count.

Answer 1

If you want to use /proc filesystem for the total counts (and not for per second counts), it is quite easy.

This works also on quite old kernels (tested on Debian Squeeze 2.6.32 kernel).

# cat /proc/1979/io
rchar: 111195372883082
wchar: 10424431162257
syscr: 130902776102
syscw: 6236420365
read_bytes: 2839822376960
write_bytes: 803408183296
cancelled_write_bytes: 374812672

For system-wide, just sum the numbers from all processes, which however will be good enough only in short-term, because as processes die, their statistics are removed from memory. You would need process accounting enabled to save them.

Meaning of these files is documented in the kernel sources file Documentation/filesystems/proc.txt:

rchar - I/O counter: chars read

The number of bytes which this task has caused to be read from storage. This is simply the sum of bytes which this process passed to read() and pread(). It includes things like tty IO and it is unaffected by whether or not actual physical disk IO was required (the read might have been satisfied from pagecache)

wchar - I/O counter: chars written

The number of bytes which this task has caused, or shall cause to be written to disk. Similar caveats apply here as with rchar.

syscr - I/O counter: read syscalls

Attempt to count the number of read I/O operations, i.e. syscalls like read() and pread().

syscw - I/O counter: write syscalls

Attempt to count the number of write I/O operations, i.e. syscalls like write() and pwrite().

read_bytes - I/O counter: bytes read

Attempt to count the number of bytes which this process really did cause to be fetched from the storage layer. Done at the submit_bio() level, so it is accurate for block-backed filesystems.

write_bytes - I/O counter: bytes written

Attempt to count the number of bytes which this process caused to be sent to the storage layer. This is done at page-dirtying time.

cancelled_write_bytes

The big inaccuracy here is truncate. If a process writes 1MB to a file and then deletes the file, it will in fact perform no writeout. But it will have been accounted as having caused 1MB of write. In other words: The number of bytes which this process caused to not happen, by truncating pagecache. A task can cause "negative" IO too.

Answer 2

Here is a SystemTap script that tracks the logical IO. It is based on the script at https://sourceware.org/systemtap/SystemTap_Beginners_Guide/traceiosect.html

#! /usr/bin/env stap
# traceio.stp
# Copyright (C) 2007 Red Hat, Inc., Eugene Teo <[email protected]>
# Copyright (C) 2009 Kai Meyer <[email protected]>
#   Fixed a bug that allows this to run longer
#   And added the humanreadable function
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License version 2 as
# published by the Free Software Foundation.
#

global reads, writes

probe vfs.read.return {
  if ($return > 0) {
    reads += $return
  }
}

probe vfs.write.return {
  if ($return > 0) {
    writes += $return
  }
}

function humanreadable(bytes) {
  if (bytes > 1024*1024*1024) {
    return sprintf("%d GiB", bytes/1024/1024/1024)
  } else if (bytes > 1024*1024) {
    return sprintf("%d MiB", bytes/1024/1024)
  } else if (bytes > 1024) {
    return sprintf("%d KiB", bytes/1024)
  } else {
    return sprintf("%d   B", bytes)
  }
}

probe timer.s(1) {
  printf("reads: %12s writes: %12s\n", humanreadable(reads), humanreadable(writes))
  # Note we don't zero out reads and writes,
  # so the values are cumulative since the script started.
}