iostat and NVMe block drivers for Intel PCIe SSD – what you need to know

This blog is relative to Linux 2.6 kernel architectures and higher

iostat is arguably the most popular monitor for I/O that exists for *nix based systems, and it certainly has a long history since it goes back to pre-Linux days. iostat provides a great summarization of the driver layer data, and even blends in CPU statistics to give you more information at a system level. But remember it is a monitor and not a debugger or tracer, that would require strace (system calls) or blktrace (block subsystem) tracers. iostat is the best tool to figure out what is happening on a drive or across a logical set of drives over the course of a workload run. Especially when you want to save off data in Excel or use a parser and plotter tool, which gives you something that is repeatable.

Let’s look at a layer diagram to understand where the iostat monitor receives data. Where data comes from is especially important in the layered world of I/O and networks. The arrow in the figure below shows the location where data is collected and here we use the nvme block driver location as an example. iostat will also optionally provide some CPU statistics which makes it more valuable as a system level monitor for your I/O constrained workloads.

This diagram provides a Layer diagram that depicts the NVMe driver and where iostat pulls data. It gives you a good picture of where you are pulling data.

blockdriver-iostat.png

How might you run iostat?

Two modes exist, the simple “on the fly” mode is just to use extended disk stats to see what might be wrong, it’s the simple approach. You can add the –t switch to get critical timestamps especially when you compare one thing to another, and start collecting data into files below.

iostat –x –m –t <device name> 5

An accurate syntax example for nvme since we are no longer in scsi land is here:
iostat –xtm /dev/nvme0n1 5

-x is for extended disk statistics

-t timestamp the monitoring records

-m is for megabytes, or k for kilobytes since storage is so capable now, especially high bandwidth PCIe SSD’s

5 is the second interval of how the data is provided to the terminal window

So maybe for longer scripted collections you should try something like this for your scripting.

iostat –xtm /dev/nvme0n1 1 <add a loop iterator here>  >  `date +%b%d-%H%M`.iostat.out &

Now you have the extended statistics plus a timestamp to work from for a large analysis in Excel.

Now if you want to know where the raw driver stats that iostat reads from, those are posted to a file called “stat”. So an example of this file is: /sys/block/nvme0n1/stat

The 11 columns from left to right in the stat file are the following and this information comes from the kernel.org document that our device driver team here at Intel follows:

Name units         description

---- -----         -----------

read I/Os requests      number of read I/Os processed

read merges requests      number of read I/Os merged with in-queue I/O

read sectors sectors       number of sectors read

read ticks milliseconds  total wait time for read requests

write I/Os requests      number of write I/Os processed

write merges requests      number of write I/Os merged with in-queue I/O

write sectors sectors       number of sectors written

write ticks milliseconds  total wait time for write requests

in_flight requests      number of I/Os currently in flight

io_ticks milliseconds  total time this block device has been active

time_in_queue   milliseconds  total wait time for all requests

I hope this helps you figure out your drive and what it’s doing. Thanks for reading.

Published on Categories Archive

About Frank Ober

Frank Ober is a Data Center Solutions Architect in the Non-Volatile Memory Group of Intel. He joined 3 years back to delve into use cases for the emerging memory hierarchy after a 25 year Enterprise Applications IT career, spanning, SAP, Oracle, Cloud Manageability and other domains. He regularly tests and benchmarks Intel SSDs against application and database workloads, and is responsible for many technology proof point partnerships with Intel software vendors.