Reviewing performance metrics on a NetApp Data ONTAP operating system can be achieved a number of ways. Recently I was reviewing some FAS2040 filers which appeared to be struggling with the load placed on them.
My first action was to SSH into each filer and use the ‘statit’ command to gather some performance data. There are a lot of commands which we can use in Data ONTAP but I find the ‘statit’ option tends to give me everything I need for a quick glance.
Please note – if you wish to follow the steps that we will be elevating our session to privileged mode – you should be careful when running like this as there is a greater risk of causing problems if you enter a command incorrectly.
Enter Privileged Mode
First we need to enter privileged mode on the filer. Connect via your SSH client of choice and then enter the command ‘priv set advanced’ –
FILER01> priv set advanced Warning: These advanced commands are potentially dangerous; use them only when directed to do so by NetApp personnel.
Run ‘statit’
Now that we are in privileged mode (denoted by the *) we can run the ‘statit’ command.
FILER01*> statit statit: expected '-b' usage: statit -b take beginning snapshot of performance stats statit -e <flags> take ending snapshot of stats and print report optional flags include: -n append 'nfsstat -c' output to the report -r report counts over interval instead of per-second rates FILER01*>
There are a few options available however the main switches are –b and –e.
The –b switch will begin a statit monitoring process while the –e switch will end the running process and generate the results output.
Here is the output from one of the filers, it is rather lengthy. If you take a look at the disk statistics and the ‘ut %’ value you will see that the active disks in aggregate 0 are all hitting 97% and above utilisation. This is incredibly high and is the result of load beyond the capability of this hardware configuration.
FILER04*>statit -e Hostname: Filer4 ID: 01234567891 Memory: 2862 MB NetApp Release 8.1.2P3 7-Mode: Wed Feb 20 19:56:15 PST 2013 <5O> Start time: Thu Jul 30 10:52:33 BST 2015 CPU Statistics 784.181888 time (seconds) 100 % 689.103174 system time 88 % 22.282110 rupt time 3 % (10779740 rupts x 2 usec/rupt) 666.821064 non-rupt system time 85 % 879.260600 idle time 112 % 657.598533 time in CP 84 % 100 % 18.613492 rupt time in CP 3 % (8927320 rupts x 2 usec/rupt) Multiprocessor Statistics (per second) cpu0 cpu1 total sk switches 42680.68 35752.68 78433.36 hard switches 28714.09 21668.33 50382.43 domain switches 1013.76 761.71 1775.48 CP rupts 6983.22 4401.03 11384.25 nonCP rupts 1385.67 976.56 2362.23 IPI rupts 0.00 0.00 0.00 grab kahuna 1.05 0.70 1.75 grab kahuna usec 69.94 44.14 114.08 CP rupt usec 18048.92 5687.28 23736.19 nonCP rupt usec 3396.29 1281.98 4678.27 idle 620195.49 501050.25 1121245.74 kahuna 4173.32 344299.48 348472.80 storage 55995.75 52585.98 108581.74 exempt 71790.87 35330.06 107120.94 raid 110495.63 13297.99 123793.63 target 24218.25 14651.52 38869.78 dnscache 0.00 0.00 0.00 cifs 117.25 81.15 198.40 wafl_exempt 0.00 0.00 0.00 wafl_xcleaner 0.00 0.00 0.00 sm_exempt 35.10 25.01 60.11 cluster 0.00 0.00 0.00 protocol 0.00 0.00 0.00 nwk_exclusive 0.00 0.00 0.00 nwk_exempt 0.00 0.00 0.00 nwk_legacy 85264.00 27207.19 112471.20 hostOS 6269.11 4502.09 10771.20 459.239782 seconds with one or more CPUs active ( 59%) 249.939725 seconds with one CPU active ( 32%) 209.300057 seconds with both CPUs active ( 27%) Domain Utilization of Shared Domains (per second) 0.00 idle 0.00 kahuna 0.00 storage 0.00 exempt 0.00 raid 0.00 target 0.00 dnscache 0.00 cifs 0.00 wafl_exempt 0.00 wafl_xcleaner 0.00 sm_exempt 0.00 cluster 0.00 protocol 0.00 nwk_exclusive 0.00 nwk_exempt 0.00 nwk_legacy 0.00 hostOS Miscellaneous Statistics (per second) 50382.43 hard context switches 0.00 NFS operations 0.00 CIFS operations 0.00 HTTP operations 7980.60 network KB received 47469.84 network KB transmitted 60638.73 disk KB read 14909.97 disk KB written 5852.86 NVRAM KB written 0.00 nolog KB written 9795.69 WAFL bufs given to clients 0.00 checksum cache hits ( 0%) 0.00 no checksum - partial buffer 0.00 FCP operations 398.44 iSCSI operations WAFL Statistics (per second) 13.14 name cache hits ( 87%) 1.90 name cache misses ( 13%) 90684.32 buf hash hits ( 74%) 31351.81 buf hash misses ( 26%) 1607.05 inode cache hits ( 100%) 0.58 inode cache misses ( 0%) 23911.16 buf cache hits ( 95%) 1382.27 buf cache misses ( 5%) 1031.22 blocks read 11251.30 blocks read-ahead 1707.74 chains read-ahead 1167.63 dummy reads 5032.45 blocks speculative read-ahead 3115.56 blocks written 10.05 stripes written 18.20 blocks page flipped 0.00 blocks over-written 0.00 wafl_timer generated CP 0.00 snapshot generated CP 0.00 wafl_avail_bufs generated CP 0.10 dirty_blk_cnt generated CP 0.00 full NV-log generated CP 0.00 back-to-back CP 0.00 flush generated CP 0.00 sync generated CP 0.00 deferred back-to-back CP 0.00 container-indirect-pin CP 0.00 low mbufs generated CP 0.00 low datavecs generated CP 4147.06 non-restart messages 365.98 IOWAIT suspends 38339029.72 next nvlog nearly full msecs 0.00 dirty buffer susp msecs 0.00 nvlog full susp msecs 521728 buffers RAID Statistics (per second) 613.61 xors 0.00 long dispatches [0] 0.00 long consumed [0] 0.00 long consumed hipri [0] 0.00 long low priority [0] 0.00 long high priority [0] 0.00 long monitor tics [0] 0.00 long monitor clears [0] 0.00 long dispatches [1] 0.00 long consumed [1] 0.00 long consumed hipri [1] 0.00 long low priority [1] 0.00 long high priority [1] 0.00 long monitor tics [1] 0.00 long monitor clears [1] 18 max batch 5.74 blocked mode xor 214.23 timed mode xor 3.17 fast adjustments 2.51 slow adjustments 0 avg batch start 0 avg stripe/msec 2970.11 checksum dispatches 68500.93 checksum consumed 10.25 tetrises written 0.00 master tetrises 0.00 slave tetrises 315.11 stripes written 299.16 partial stripes 15.96 full stripes 3096.44 blocks written 1477.70 blocks read 2.64 1 blocks per stripe size 14 2.37 2 blocks per stripe size 14 3.58 3 blocks per stripe size 14 5.77 4 blocks per stripe size 14 9.38 5 blocks per stripe size 14 13.89 6 blocks per stripe size 14 20.62 7 blocks per stripe size 14 28.19 8 blocks per stripe size 14 38.10 9 blocks per stripe size 14 45.62 10 blocks per stripe size 14 49.00 11 blocks per stripe size 14 46.33 12 blocks per stripe size 14 33.67 13 blocks per stripe size 14 15.96 14 blocks per stripe size 14 Network Interface Statistics (per second) iface side bytes packets multicasts errors collisions pkt drops e0a recv 1630779.37 2453.26 0.29 0.00 0.00 xmit 9933003.43 7344.30 0.00 0.00 0.00 e0b recv 1659188.76 2869.79 0.29 0.00 0.00 xmit 9920787.03 7348.96 0.00 0.00 0.00 e0c recv 1615344.85 2167.53 0.29 0.00 0.00 xmit 9692776.40 7170.42 0.00 0.00 0.00 e0d recv 1639064.10 2903.87 0.00 0.00 0.00 xmit 9532307.16 7056.18 0.01 0.00 0.00 e0P recv 27.83 0.29 0.03 0.00 0.00 xmit 20.41 0.26 0.00 0.00 0.00 vh recv 0.00 0.00 0.00 0.00 0.00 xmit 0.00 0.00 0.00 0.00 0.00 e0d-4 recv 2475.21 21.34 8.72 0.00 0.00 xmit 2240.23 11.47 0.02 0.00 0.00 e0d-202 recv 1625250.71 2882.29 0.29 0.00 0.00 xmit 9527982.49 7043.35 0.01 0.00 0.00 Disk Statistics (per second) ut% is the percent of time the disk was busy. xfers is the number of data-transfer commands issued per second. xfers = ureads + writes + cpreads + greads + gwrites chain is the average number of 4K blocks per command. usecs is the average disk round-trip time per 4K block. disk ut% xfers ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs /aggr0/plex0/rg0: 0c.00.0 7 8.77 0.34 1.21 2214 6.39 24.06 325 2.03 8.82 697 0.00 .... . 0.00 .... . 0c.00.2 7 8.96 0.34 1.21 2924 6.61 23.32 333 2.00 8.58 786 0.00 .... . 0.00 .... . 0c.00.4 99 120.93 105.98 4.57 13016 9.15 11.48 4007 5.80 8.30 5126 0.00 .... . 0.00 .... . 0d.01.2 98 122.78 108.25 4.45 11563 9.01 11.70 3886 5.52 8.67 3980 0.00 .... . 0.00 .... . 0c.00.6 99 121.07 106.84 4.53 13546 8.79 11.94 3855 5.44 8.76 4700 0.00 .... . 0.00 .... . 0d.02.1 97 120.84 106.81 4.57 11111 8.64 12.28 3610 5.39 8.83 3929 0.00 .... . 0.00 .... . 0d.01.5 97 122.65 108.30 4.51 11127 8.79 11.94 3768 5.56 8.64 3924 0.00 .... . 0.00 .... . 0c.00.8 98 120.33 106.59 4.50 12804 8.50 12.58 3514 5.24 8.79 4371 0.00 .... . 0.00 .... . 0d.02.2 97 121.99 107.67 4.50 11125 8.80 12.00 3761 5.52 8.64 3854 0.00 .... . 0.00 .... . 0d.01.14 97 122.03 107.78 4.48 11114 8.73 12.06 3715 5.51 8.85 3656 0.00 .... . 0.00 .... . 0c.00.10 99 121.55 107.34 4.48 13788 8.78 11.98 4016 5.43 8.76 4758 0.00 .... . 0.00 .... . 0d.03.11 98 121.75 107.07 4.55 12406 9.14 11.57 4739 5.54 8.61 4662 0.00 .... . 0.00 .... . 0d.03.0 97 115.83 103.16 4.58 11392 7.65 14.27 3607 5.02 8.70 4601 0.00 .... . 0.00 .... . 0d.02.4 97 116.66 103.97 4.53 11736 7.75 14.30 3663 4.93 8.56 4597 0.00 .... . 0.00 .... . 0d.03.1 97 116.37 103.58 4.61 11350 7.82 14.12 3726 4.97 8.77 4403 0.00 .... . 0.00 .... . 0d.02.21 95 116.32 103.65 4.60 10293 7.82 14.07 3154 4.85 8.92 3775 0.00 .... . 0.00 .... . /aggr0/plex0/rg1: 0d.02.9 6 11.17 0.00 2.50 2400 5.30 30.64 317 5.87 27.73 118 0.00 .... . 0.00 .... . 0d.01.17 6 11.20 0.00 2.50 2000 5.34 30.43 317 5.86 27.73 124 0.00 .... . 0.00 .... . 0d.02.10 97 121.82 106.68 4.50 11011 9.11 12.19 3549 6.03 8.65 3687 0.00 .... . 0.00 .... . 0d.01.20 97 123.83 108.03 4.48 11384 9.70 11.76 3572 6.11 8.11 3736 0.00 .... . 0.00 .... . 0d.02.11 98 124.13 108.45 4.49 11474 9.55 11.82 3751 6.12 8.35 3864 0.00 .... . 0.00 .... . 0d.01.22 98 124.74 108.73 4.49 11462 9.83 11.50 3756 6.18 8.24 3713 0.00 .... . 0.00 .... . 0d.02.13 97 122.76 107.26 4.50 11360 9.41 11.76 3611 6.08 8.49 3652 0.00 .... . 0.00 .... . 0d.02.14 97 122.80 107.16 4.47 11460 9.52 12.01 3609 6.12 8.21 3972 0.00 .... . 0.00 .... . 0d.02.15 97 123.81 108.27 4.51 11161 9.46 11.99 3518 6.09 8.21 3722 0.00 .... . 0.00 .... . 0d.02.16 97 122.59 106.95 4.49 11017 9.50 11.90 3665 6.14 8.07 3949 0.00 .... . 0.00 .... . 0d.02.20 97 121.95 106.35 4.50 10792 9.50 11.85 3572 6.10 8.43 3692 0.00 .... . 0.00 .... . 0d.03.2 97 117.37 103.62 4.57 11744 8.19 14.10 3718 5.57 8.76 4368 0.00 .... . 0.00 .... . 0d.02.22 96 116.45 102.43 4.58 10380 8.44 14.01 3092 5.57 8.29 3735 0.00 .... . 0.00 .... . 0d.03.3 97 116.64 102.85 4.58 11419 8.16 14.36 3546 5.62 8.50 4152 0.00 .... . 0.00 .... . 0d.02.23 95 117.70 103.61 4.63 9757 8.36 13.96 3087 5.73 8.46 3609 0.00 .... . 0.00 .... . 0d.03.4 97 117.20 103.17 4.60 11292 8.35 13.98 3564 5.67 8.36 4322 0.00 .... . 0.00 .... . Aggregate statistics: Minimum 6 8.77 0.00 5.30 2.00 0.00 0.00 Mean 86 106.72 92.85 8.44 5.43 0.00 0.00 Maximum 99 124.74 108.73 9.83 6.18 0.00 0.00 Spares and other disks: 0a.18 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0a.20 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0a.17 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0a.29 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0a.23 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.02.8 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0a.24 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0a.28 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0a.22 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0a.27 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0a.21 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0a.26 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0a.25 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.01.0 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.01.1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.01.3 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.01.4 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.01.6 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.01.7 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.01.9 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.01.8 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.01.10 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.01.11 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.01.13 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.01.12 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.01.15 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.01.16 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.01.18 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.01.19 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.01.21 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.01.23 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.02.0 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.02.3 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.02.5 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.02.7 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.02.6 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0a.19 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.02.12 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.02.18 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.02.17 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.02.19 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.03.6 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.03.5 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.03.8 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.03.9 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.03.7 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.03.10 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.03.13 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.03.12 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.03.15 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.03.14 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.03.17 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.03.18 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.03.19 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.03.16 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.03.20 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.03.21 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.03.23 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0d.03.22 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0c.00.1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0c.00.3 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0c.00.5 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0c.00.7 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0c.00.9 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0c.00.11 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0a.16 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . FCP Statistics (per second) 0.00 FCP Bytes recv 0.00 FCP Bytes sent 0.00 FCP ops iSCSI Statistics (per second) 5872949.31 iSCSI Bytes recv 37452989.27 iSCSI Bytes xmit 398.44 iSCSI ops Interrupt Statistics (per second) 3222.95 int_1 0.18 int_2 1846.67 Gigabit Ethernet (IRQ 4) 1299.32 Gigabit Ethernet (IRQ 5) 1846.61 Gigabit Ethernet (IRQ 6) 1530.99 Gigabit Ethernet (IRQ 7) 3222.95 int_8 3222.95 int_9 0.00 RTC 0.00 IPI 999.90 Msec Clock 17192.53 total NVRAM Statistics (per second) 0.00 total dma transfer KB 0.00 wafl write req data KB 0.00 dma transactions 0.00 dma destriptors 269.21 waitdone preempts 0.00 waitdone delays 0.00 transactions not queued 141.52 transactions queued 141.59 transactions done 22.37 total waittime (MS) 481.23 completion wakeups 93.19 nvdma completion wakeups 17.81 nvdma completion waitdone 5852.92 total nvlog KB 0.00 nvlog shadow header array full 0.00 channel1 dma transfer KB 0.00 channel1 dma transactions 0.00 channel1 dma descriptors
Exit Privileged Mode
When we are finished with the statit command it is sensible to return to the default permission level by entering the command ‘priv set’.
FILER01*> priv set FILER01>
We know the CLI has returned to the default mode as the * is no longer displayed.
I find it best to copy the data into something like Notepad++ so I can easily manipulate and view the data outside my SSH session. There are of course many tools and commands for monitoring and troubleshooting performance but often ‘statit’ gives me exactly what I need without having to rely on a customer having deployed a monitoring solution in advance.