Logo Questions Linux Laravel Mysql Ubuntu Git Menu
HTML CSS JAVASCRIPT SQL PYTHON PHP BOOTSTRAP JAVA JQUERY R React Kotlin
 

Interpreting data from /proc/[id]/mountstats or /proc/self/mountstats

Tags:

linux

bash

proc

I would like to monitor the IOs of my NFS shares. There are plenty of programs which are interpreting the data within /proc/[id]/mountstats ([id] = id of nfs client/server) or /proc/self/mountstats. The problem is, these programs calculate some IO per second. This is very nice in the command line, but it is quite useless for my monitoring solution. So I want to interpret the counters of mountstats myself. My problem now is, that I need some documentation of mountstats for nfs v4.

So far my search haven't brought up any useful.

Thanks for your help

sample output of /proc/self/mountstats on Ubuntu (nfs client):

...
device IPADRESS:/mountpoint mounted on /mnt/mountpoint with fstype nfs4 statvers=1.0
    opts:   rw,vers=4,rsize=1048576,wsize=1048576,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,nointr,proto=tcp,timeo=600,retrans=2,sec=sys
    age:    77663
    caps:   caps=0x1e,wtmult=512,dtsize=4096,bsize=0,namelen=255
    nfsv4:  bm0=0xfdffbfff,bm1=0xf9be3e,acl=0x3
    sec:    flavor=1,pseudoflavor=1
    events: 1462723 25756220 1025148 1164028 1083099 2027020 31347107 1984730 279773 1915700 1346583 21077199 520498 31825 525298 39026 2180470 1083041 0 16822 990144 11011 0 0 0
    bytes:  46686144592 3990939437 0 0 30238687174 3759616491 7739053 1346583
    RPC iostats version: 1.0  p/v: 100003/4 (nfs)
    xprt:   tcp 852 1 2 0 0 11822378 11822377 0 17669710 13357205
    per-op statistics
            NULL: 0 0 0 0 0 0 0 0
            READ: 1950114 1950114 0 330348288 30355844804 105050 6678050 6907640
           WRITE: 911822 911822 0 3932410080 215189360 10653830 1472320 12211620
          COMMIT: 389509 389509 0 65718096 88808052 8060 253550 284360
            OPEN: 1168845 1168845 0 292577612 554554444 430040 628900 1229990
    OPEN_CONFIRM: 43 43 0 7048 2924 0 10 10
     OPEN_NOATTR: 0 0 0 0 0 0 0 0
    OPEN_DOWNGRADE: 54 54 0 9952 12744 90 0 120
           CLOSE: 1081161 1081161 0 191495392 255153996 201030 404620 767870
         SETATTR: 43303 43303 0 8753024 10046296 1060 25650 30380
          FSINFO: 2 2 0 268 184 0 0 0
           RENEW: 0 0 0 0 0 0 0 0
     SETCLIENTID: 0 0 0 0 0 0 0 0
    SETCLIENTID_CONFIRM: 0 0 0 0 0 0 0 0
            LOCK: 1140050 1140050 0 246277244 77523400 76270 326950 469890
           LOCKT: 1 1 0 188 52 0 0 0
           LOCKU: 1037895 1037895 0 187916912 70576860 18130 357530 441480
          ACCESS: 450451 450451 0 72511716 102702828 21440 222460 280950
         GETATTR: 1462724 1462724 0 224029496 310097488 21290 390910 485350
          LOOKUP: 2043380 2043380 0 361899452 532433688 18200 499200 585170
     LOOKUP_ROOT: 1 1 0 116 232 0 0 0
          REMOVE: 32356 32356 0 5953448 7765440 600 15450 18560
          RENAME: 20388 20388 0 5593088 9215376 510 7740 9750
            LINK: 537 537 0 132024 231984 0 220 270
         SYMLINK: 0 0 0 0 0 0 0 0
          CREATE: 999 999 0 227376 293140 30 650 830
        PATHCONF: 1 1 0 140 72 0 0 0
          STATFS: 2872 2872 0 419360 333152 580 3980 4880
        READLINK: 0 0 0 0 0 0 0 0
         READDIR: 82718 82718 0 14625720 118955760 770 48470 53310
     SERVER_CAPS: 3 3 0 400 264 0 0 0
     DELEGRETURN: 0 0 0 0 0 0 0 0
          GETACL: 0 0 0 0 0 0 0 0
          SETACL: 0 0 0 0 0 0 0 0
    FS_LOCATIONS: 0 0 0 0 0 0 0 0
 ...
like image 869
tiflor Avatar asked Apr 23 '12 12:04

tiflor

People also ask

What does the mountstats command do?

The mountstats command displays various NFS client statisitics for each given mountpoint . If no mountpoint is given, statistics will be displayed for all NFS mountpoints on the client. Sub-commands Valid mountstats (8) subcommands are: mountstats Display a combination of per-op RPC statistics, NFS event counts, and NFS byte counts.

What are the valid mountstats (8) subcommands?

Sub-commands Valid mountstats (8) subcommands are: mountstats Display a combination of per-op RPC statistics, NFS event counts, and NFS byte counts. This is the default sub- command that will be executed if no sub-command is given. iostat Display iostat-like statistics. nfsstat Display nfsstat-like statistics.

How to display only the NFS and RPC Statistics in mountstats?

Options specific to the mountstats sub-command -n, --nfs Display only the NFS statistics -r, --rpc Display only the RPC statistics -R, --raw Display only the raw statistics.

How do I find the mount point of a file system?

The 2nd column reveals the mount point. The 3rd column tells the file-system type. The 4th column tells you if it is mounted read-only (ro) or read-write (rw). The 5th and 6th columns are dummy values designed to match the format used in /etc/mtab.

2 Answers

Complete source: http://www.fsl.cs.stonybrook.edu/~mchen/mountstat-format.txt

       Format of NFS mountstat

Linux kernel version: 2.6.32-358.el6.x86_64

Important files for statistics:

- linux/fs/nfs/super.c: nfs_show_stats
- linux/include/linux/nfs_iostat.h: nfs_stat_eventcounters
- linux/net/sunrpc/stats.c: rpc_print_iostats
- linux/net/sunrpc/xprtsock.c: xs_tcp_print_stats

Server Capabilities (caps)

  1. caps: server capabilities
  2. wtmult: server disk block size
  3. dtsize: readdir size
  4. bsize: server block size
  5. namlen:

nfsv4 (nfs_show_stats)

  1. bm0: attribute_bitmask[0], which attributes are supported
  2. bm1: attribute_bitmask[1], which attributes are supported
  3. acl: acl_bitmask, which acl attributes are supported

sec, security flavor in effect for this mount (nfs_show_stats)

  1. flavor: auth->au_ops->au_flavor
  2. pseudoflavor: auth->au_flavor

Events (linux/nfs_iostat.h: nfs_stat_eventcounters)

  1. inoderevalidate
  2. dentryrevalidate
  3. datainvalidate
  4. attrinvalidate
  5. vfsopen
  6. vfslookup
  7. vfsaccess
  8. vfsupdatepage
  9. vfsreadpage
  10. vfsreadpages
  11. vfswritepage
  12. vfswritepages
  13. vfsgetdents
  14. vfssetattr
  15. vfsflush
  16. vfsfsync
  17. vfslock
  18. vfsrelease
  19. congestionwait
  20. setattrtrunc
  21. extendwrite
  22. sillyrename
  23. shortread
  24. shortwrite
  25. delay
  26. pnfs_read
  27. pnfs_write

Bytes (linux/nfs_iostat.h: nfs_stat_bytecounters)

  1. normalreadbytes
  2. normalwritebytes
  3. directreadbytes
  4. directwritebytes
  5. serverreadbytes
  6. serverwritebytes (serverwrittenbytes)
  7. readpages
  8. writepages

xprt (tcp)

  • linux/net/sunrpc/xprtsock.c: xs_tcp_print_stats
  • linux/include/linux/sunrpc/xprt.h: struct rpc_xprt
  1. srcport: Ephemeral port
  2. bind_count: How many rpcbind operations
  3. connect_count: How many TCP connects
  4. connect_time: How long connects have taken
  5. idle_time: How long transport has been idle
  6. rpcsends: How many socket sends
  7. rpcrecvs: How many socket receives
  8. badxids: How many unmatchable XIDs have been received
  9. req_u: Average requests on the wire (slot table utilization)
  10. bklog_u: backlog queue utilization (average length of baklog queue)
  11. max_slots: max rpc_slots used
  12. sending_u: send q utilization
  13. pending_u: pend q utilization

Each RPC operation type shows (rpc_print_iostats)

The stats recording: rpc_exit_task ---> xprt_release ---> rpc_count_iostats

All NFS protocols are explained here: http://www.freesoft.org/CIE/Topics/115.htm

  1. The name of the op type
  2. ops How many ops of this type have been requested
  3. trans: How many transmissions of this op type have been sent
  4. timeouts: How many timeouts of this op type have occurred
  5. bytes_sent: How many bytes have been sent for this op type
  6. bytes_recv: How many bytes have been received for this op type
  7. queue: How long ops of this type have waited in queue before being transmitted (millisecond)
  8. rtt: How long the client waited to receive replies of this op type from the server (millisecond)
  9. execute: How long ops of this type take to execute (from rpc_init_task to rpc_exit_task) (millisecond)

rpc_run_task ---> rpc_new_task ---> rpc_init_task

The last three time statistics are collected using ns, but they are converted to ms when they are printed out.

The execute time should approximate to (queue + rtt).

Note, in the Linux source code, the names of the fields are prefixed with "om_", for example, the first field is "om_ops" instead of "ops".

NFSv4.0 EXAMPLE

device 130.245.177.235:/nfsdata/ mounted on /mnt with fstype nfs4 statvers=1.1
    opts:    rw,vers=4,rsize=1048576,wsize=1048576,namlen=255,acregmin=3,acregmax=60,
            acdirmin=30,acdirmax=60,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,
            clientaddr=130.245.177.234,minorversion=0,local_lock=none
    age:    19  (time, in seconds, since the NFS is mounted)
    caps:    caps=0xfff7,wtmult=512,dtsize=32768,bsize=0,namlen=255
    nfsv4:    bm0=0xfdffafff,bm1=0xf9be3e,acl=0x0
    sec:    flavor=1,pseudoflavor=1
    events:    0 168232 0 0 0 10095 217808 0 2 9797 0 9739 0 0 19739 19739 0 19739 0 0 0 0 0 0 0 0 0
    bytes:    1612840960 0 0 0 627536112 0 158076 0
    RPC iostats version: 1.0  p/v: 100003/4 (nfs)
    xprt:    tcp 737 0 1 0 0 69698 69697 0 81817 0 2 1082 12119
    per-op statistics
            NULL: 0 0 0 0 0 0 0 0
            READ: 9797 9797 0 1567520 628138612 71 7953 8200
           WRITE: 0 0 0 0 0 0 0 0
          COMMIT: 0 0 0 0 0 0 0 0
            OPEN: 19740 19740 0 4737600 7343280 505 3449 4172
    OPEN_CONFIRM: 10211 10211 0 1552072 694348 74 836 1008
     OPEN_NOATTR: 0 0 0 0 0 0 0 0
    OPEN_DOWNGRADE: 0 0 0 0 0 0 0 0
           CLOSE: 19739 19739 0 3316152 2605548 334 3045 3620
         SETATTR: 0 0 0 0 0 0 0 0
          FSINFO: 1 1 0 132 108 0 0 0
           RENEW: 0 0 0 0 0 0 0 0
     SETCLIENTID: 0 0 0 0 0 0 0 0
    SETCLIENTID_CONFIRM: 0 0 0 0 0 0 0 0
            LOCK: 0 0 0 0 0 0 0 0
           LOCKT: 0 0 0 0 0 0 0 0
           LOCKU: 0 0 0 0 0 0 0 0
          ACCESS: 96 96 0 14584 19584 0 8 10
         GETATTR: 1 1 0 132 188 0 0 0
          LOOKUP: 10095 10095 0 1655576 2382420 36 898 1072
     LOOKUP_ROOT: 0 0 0 0 0 0 0 0
          REMOVE: 0 0 0 0 0 0 0 0
          RENAME: 0 0 0 0 0 0 0 0
            LINK: 0 0 0 0 0 0 0 0
         SYMLINK: 0 0 0 0 0 0 0 0
          CREATE: 0 0 0 0 0 0 0 0
        PATHCONF: 1 1 0 128 72 0 0 0
          STATFS: 0 0 0 0 0 0 0 0
        READLINK: 0 0 0 0 0 0 0 0
         READDIR: 0 0 0 0 0 0 0 0
     SERVER_CAPS: 2 2 0 256 176 0 0 0
     DELEGRETURN: 0 0 0 0 0 0 0 0
          GETACL: 0 0 0 0 0 0 0 0
          SETACL: 0 0 0 0 0 0 0 0
    FS_LOCATIONS: 0 0 0 0 0 0 0 0
    RELEASE_LOCKOWNER: 0 0 0 0 0 0 0 0
         SECINFO: 0 0 0 0 0 0 0 0
     EXCHANGE_ID: 0 0 0 0 0 0 0 0
    CREATE_SESSION: 0 0 0 0 0 0 0 0
    DESTROY_SESSION: 0 0 0 0 0 0 0 0
        SEQUENCE: 0 0 0 0 0 0 0 0
    GET_LEASE_TIME: 0 0 0 0 0 0 0 0
    RECLAIM_COMPLETE: 0 0 0 0 0 0 0 0
       LAYOUTGET: 0 0 0 0 0 0 0 0
    GETDEVICEINFO: 0 0 0 0 0 0 0 0
    LAYOUTCOMMIT: 0 0 0 0 0 0 0 0
    LAYOUTRETURN: 0 0 0 0 0 0 0 0


NFSv4.1 EXAMPLE
===============
device 130.245.177.235:/nfsdata mounted on /mnt with fstype nfs4 statvers=1.1
    opts:   rw,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=130.245.177.234,local_lock=none
    age:    44
    impl_id:    name='',domain='',date='0,0'
    caps:   caps=0x3ffff,wtmult=512,dtsize=32768,bsize=0,namlen=255
    nfsv4:  bm0=0xfdffbfff,bm1=0xf9be3e,bm2=0x800,acl=0x3,sessions,pnfs=not configured
    sec:    flavor=1,pseudoflavor=1
    events: 67379 1692812 0 30749 0 980 1984612 0 0 0 0 0 0 0 96611 96611 0 96611 0 0 0 0 0 0 0 0 0 
    bytes:  0 0 0 0 0 0 0 0 
    RPC iostats version: 1.0  p/v: 100003/4 (nfs)
    xprt:   tcp 674 0 1 0 0 279772 279771 0 649944 0 5 1711 90409
    per-op statistics
            NULL: 0 0 0 0 0 0 0 0
            READ: 0 0 0 0 0 0 0 0
           WRITE: 0 0 0 0 0 0 0 0
          COMMIT: 0 0 0 0 0 0 0 0
            OPEN: 96612 96612 0 27824256 34007424 13349 16253 32061
    OPEN_CONFIRM: 0 0 0 0 0 0 0 0
     OPEN_NOATTR: 0 0 0 0 0 0 0 0
    OPEN_DOWNGRADE: 0 0 0 0 0 0 0 0
           CLOSE: 96610 96610 0 19708440 17003360 2936 11082 15386
         SETATTR: 0 0 0 0 0 0 0 0
          FSINFO: 1 1 0 168 152 0 0 0
           RENEW: 0 0 0 0 0 0 0 0
     SETCLIENTID: 0 0 0 0 0 0 0 0
    SETCLIENTID_CONFIRM: 0 0 0 0 0 0 0 0
            LOCK: 0 0 0 0 0 0 0 0
           LOCKT: 0 0 0 0 0 0 0 0
           LOCKU: 0 0 0 0 0 0 0 0
          ACCESS: 16932 16932 0 3183200 2844576 83 1919 2308
         GETATTR: 67380 67380 0 12128380 15632160 336 8367 10644
          LOOKUP: 1236 1236 0 247176 167784 7 153 197
     LOOKUP_ROOT: 0 0 0 0 0 0 0 0
          REMOVE: 0 0 0 0 0 0 0 0
          RENAME: 0 0 0 0 0 0 0 0
            LINK: 0 0 0 0 0 0 0 0
         SYMLINK: 0 0 0 0 0 0 0 0
          CREATE: 980 980 0 219516 305760 5 168 197
        PATHCONF: 1 1 0 164 116 0 0 0
          STATFS: 0 0 0 0 0 0 0 0
        READLINK: 0 0 0 0 0 0 0 0
         READDIR: 0 0 0 0 0 0 0 0
     SERVER_CAPS: 2 2 0 328 280 0 0 0
     DELEGRETURN: 0 0 0 0 0 0 0 0
          GETACL: 0 0 0 0 0 0 0 0
          SETACL: 0 0 0 0 0 0 0 0
    FS_LOCATIONS: 0 0 0 0 0 0 0 0
    RELEASE_LOCKOWNER: 0 0 0 0 0 0 0 0
         SECINFO: 0 0 0 0 0 0 0 0
     EXCHANGE_ID: 0 0 0 0 0 0 0 0
    CREATE_SESSION: 0 0 0 0 0 0 0 0
    DESTROY_SESSION: 0 0 0 0 0 0 0 0
        SEQUENCE: 0 0 0 0 0 0 0 0
    GET_LEASE_TIME: 0 0 0 0 0 0 0 0
    RECLAIM_COMPLETE: 0 0 0 0 0 0 0 0
       LAYOUTGET: 0 0 0 0 0 0 0 0
    GETDEVICEINFO: 0 0 0 0 0 0 0 0
    LAYOUTCOMMIT: 0 0 0 0 0 0 0 0
    LAYOUTRETURN: 0 0 0 0 0 0 0 0
    SECINFO_NO_NAME: 0 0 0 0 0 0 0 0
    TEST_STATEID: 0 0 0 0 0 0 0 0
    FREE_STATEID: 0 0 0 0 0 0 0 0
    GETDEVICELIST: 0 0 0 0 0 0 0 0
    BIND_CONN_TO_SESSION: 0 0 0 0 0 0 0 0
    DESTROY_CLIENTID: 0 0 0 0 0 0 0 0


# vim: tabstop=2:shiftwidth=2:expandtab:foldmethod=indent

Also https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/include/linux/nfs_iostat.h and https://utcc.utoronto.ca/~cks/space/blog/linux/NFSMountstatsIndex for additional reading.

like image 195
GP92 Avatar answered Nov 15 '22 16:11

GP92

Apparently there is no better "documentation" than the code of the mountstats utility.

like image 24
Armali Avatar answered Nov 15 '22 15:11

Armali
Sign in to Comment

Related questions

Linux - securing environment variables
Is there a way to restore/recover nohup to see the output in the console?
Is there a python module to parse Linux's sysfs?
Linux C++ project directory layout - CMake
linux: suspend process at startup
sed with special characters
Static global variable and static local variable in driver function
setjmp and longjmp - understanding with examples
Quick way to email results of a mysql query
What API of Linux/Windows corresponds to limiting network (download/upload) speed?
How to use a global variable in gcc inline assembly
Shrink window in GTK+ dynamically when content shrinks?
file not found after mysql export
Java <-> C Bridge
Good books for learning C++ programming on UNIX (pref Linux)? [closed]
OpenMP threads executing on the same cpu core
Do I need a PORT when joining a multicast group or just the IP?
Read unbuffered data from pipe in Perl
Difference between static library and relocatable object file?
can we use poll function with unnamed pipes?

Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!