This blog opens a whole series of the handy instructions for benchmarking SSDs, from the ground up hardware configuration, up to the software stack. Here I’ll be writing only about hardware, CPUs, platform, BIOS options, etc. You’ll be surprised how varied the results of those options are especially for Intel Optane drives.
What’s special in benchmarking them? Well, you know the answer, the latency under small queue depth is special, out of 10us latency, the media contribution is small, while the NVMe protocol and SW stack contribute a significant portion. And so, if we speak about the software, then the CPU performance matters the most… Does it? yes...
Alright, so, assuming we’re benchmarking P4800X drive, which is a server product, and so we come up with a server platform. Let’s take an off-the-shelf dual socket server, in my example, Intel R2208WFTZS. How should I configure it? The easiest answer – based on your needs, or the needs of your application.
The details I’m going to share are in the area of the extreme tuning, which is not necessarily a production environment. My focus here is the best storage performance, nothing but that. As we agreed on that, and I’d like to demonstrate lowest latency of the Optane drive which is according to specs is 10us.
In a highly tuned environment, I am able to achieve sub 7us. What do I need for that? First, highest clock rate CPU – as the software impact is significant, moving to a higher frequency will help to reduce it. So, for my platform, I’ll use Intel® Xeon® Gold 6154 Processor.
Base frequency is 3.0GHz, and Turbo is 3.7GHz, - that’s just wow!
I usually start with the “Power profile” setting in the BIOS configuration, which is based on the motherboard vendor setups of some preconfigured options.
This is an easy start just to toggle all performance as seen in the view below:
Do I need Turbo or not? Well, depends on the goal, if we’re looking for the lowest average latency on single threaded QD=1 workloads, Turbo will definitely help. But if I need lowest maximum latency, which can be disturbed by the CPU and the OS power switching time, it's better to disable it. e.. It is recommended to explore both options and choose the one that best fits your system/configuration.
In the same way, I’ll disable C-States, which are CPU power states as they will hit both average and maximum latency. And so that means, for the benchmarking, I’ll likely disable all CPU power management under Enhanced Intel SpeedStep® Technology—that’s not just that, Hyper-Threading is another option.
For the ultimate benchmark condition, we want to have a full control over the CPU core and not share it between two logical cores. So, let’s disable it too.
Once that’s done, additional system power consumption is expected as we disabled all power saving options. We need to be sure that server cooling subsystem is also capable to deliver that. For this particular platform, there is an option which switches fans between acoustic and performance mode.
I went to performance as seen in the screenshot below:
And finally, make sure you understand what PCIe slot you put the drive in. I can be connected to CPU0 or CPU1 (in case of dual socket architecture) and routed directly or through PCH or PCIe switch. This will be very important during OS and benchmark tool configuration and set up of CPU affinity.
The following Linux utility will help you to visualize your current PCIe layout, which is part of the “hwloc” package: # lstopo -v --of png> /tmp/numa.png
In the PCI layout topology view below, the platform has two P4800X drives connected (8086:2701) to different CPUs.
To summarize, make sure your HW configuration is optimal for the needs of the test and BIOS is configured accordantly. Disable options which could disturb the latency!
Read the next blog, "Tuning the performance of Intel Optane SSDs on Linux Operating Systems".