This blog is relative to Linux 2.6 kernel architectures and higher
iostat is arguably the most popular monitor for I/O that exists for *nix based systems, and it certainly has a long history since it goes back to pre-Linux days. iostat provides a great summarization of the driver layer data, and even blends in CPU statistics to give you more information at a system level. But remember it is a monitor and not a debugger or tracer, that would require strace (system calls) or blktrace (block subsystem) tracers. iostat is the best tool to figure out what is happening on a drive or across a logical set of drives over the course of a workload run. Especially when you want to save off data in Excel or use a parser and plotter tool, which gives you something that is repeatable.
Let’s look at a layer diagram to understand where the iostat monitor receives data. Where data comes from is especially important in the layered world of I/O and networks. The arrow in the figure below shows the location where data is collected and here we use the nvme block driver location as an example. iostat will also optionally provide some CPU statistics which makes it more valuable as a system level monitor for your I/O constrained workloads.
This diagram provides a Layer diagram that depicts the NVMe driver and where iostat pulls data. It gives you a good picture of where you are pulling data.
How might you run iostat?
Two modes exist, the simple “on the fly” mode is just to use extended disk stats to see what might be wrong, it’s the simple approach. You can add the –t switch to get critical timestamps especially when you compare one thing to another, and start collecting data into files below.
iostat –x –m –t <device name> 5
An accurate syntax example for nvme since we are no longer in scsi land is here:
iostat –xtm /dev/nvme0n1 5
-x is for extended disk statistics
-t timestamp the monitoring records
-m is for megabytes, or k for kilobytes since storage is so capable now, especially high bandwidth PCIe SSD’s
5 is the second interval of how the data is provided to the terminal window
So maybe for longer scripted collections you should try something like this for your scripting.
iostat –xtm /dev/nvme0n1 1 <add a loop iterator here> > `date +%b%d-%H%M`.iostat.out &
Now you have the extended statistics plus a timestamp to work from for a large analysis in Excel.
Now if you want to know where the raw driver stats that iostat reads from, those are posted to a file called “stat”. So an example of this file is: /sys/block/nvme0n1/stat
The 11 columns from left to right in the stat file are the following and this information comes from the kernel.org document that our device driver team here at Intel follows:
Name units description
---- ----- -----------
read I/Os requests number of read I/Os processed
read merges requests number of read I/Os merged with in-queue I/O
read sectors sectors number of sectors read
read ticks milliseconds total wait time for read requests
write I/Os requests number of write I/Os processed
write merges requests number of write I/Os merged with in-queue I/O
write sectors sectors number of sectors written
write ticks milliseconds total wait time for write requests
in_flight requests number of I/Os currently in flight
io_ticks milliseconds total time this block device has been active
time_in_queue milliseconds total wait time for all requests
I hope this helps you figure out your drive and what it’s doing. Thanks for reading.