Posted on 13/06/2017 by Helen Koike
When I last wrote about NVMe, the feature to improve NVMe performance over emulated environments was just a living discussion and a work in progress patch. However, it has now been officially released in the NVMe Specification Revision 1.3 under the name "Doorbell Buffer Config command", along with an implementation that is already in the mainline Linux kernel! \o/
You can already feel the difference in performance if you compile Kernel 4.12-rc1 (or later) and run it over a virtual machine hosted on Google Compute Engine. Google actually updated their hypervisor as soon as the feature was ratified by the NVMe working group, even before it was publicly released.
There were very few changes from the original proposal, I.e. opcodes, return values and now fancy names; the buffers (as described in my last post) are now called Shadow Doorbell and EventIdx buffers.
In short, the first one mimics the Doorbell registers in memory, allowing the emulated controller to fetch the Doorbell value when convenient instead of waiting for the Doorbell register to be written. For its part, the EventIdx provides a hint given by the emulated controller to tell the host if the Doorbell register needs to be updated (in case the emulated controller is not fetching the Doorbell value from the Shadow Doorbell buffer). You can check section 7.13 of the specification for an example of usage.
The following test results were obtained in a machine of type n1-standard-4 (4 vCPUs, 15 GB memory) at Google Cloud Engine platform with Kernel 4.12.0-rc5 using the following command:
$ sudo fio --time_based --name=benchmark --runtime=30 \ --filename=/dev/nvme0n1 --nrfiles=1 --ioengine=libaio --iodepth=32 \ --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=1 \ --rw=randread --blocksize=4k --randrepeat=0
Results (in Input/Ouput Operations per Second):
Without Shadow Doorbell and EventIdx buffers: 43.9K IOPS
With Shadow Doorbell and EventIdx buffers: 184K IOPS
Gain ~= 4 times
Screenshot - Without Shadow Doorbell and EventIdx buffers
Screenshot - With Shadow Doorbell and EventIdx buffers
Enjoy your enhanced numbers of IOPS! :D
18/05/2018
Earlier this month, Olivier Crête, Nicolas Dufresne, George Kiagiadakis & I attended the GStreamer Spring Hackfest in Lund, Sweden. Hosted…
09/05/2018
A few months ago, Robert Foss wrote a blog post about virtualizing GPU Access. Here's a look at some of the major improvements that have…
01/05/2018
Generously hosted by Axis in the beautiful Lund, Sweden, the annual spring hackfest is an occasion for the community to get together to…
17/04/2018
The i.MX 6 platform has for the past few years enjoyed a large effort to add upstream support to Linux and surrounding projects. The newly…
23/03/2018
Following on from part 1 in the series, part 2 covers more developments in low-level graphics, including support for buffer modifiers in…
20/03/2018
The latest enhancements to the DRM subsystem have made mainline Linux much more attractive, making drivers easier to write, applications…
Whether writing a line of code or shaping a longer-term strategic software development plan, we'll help you navigate the ever-evolving world of Open Source.
Comments (0)
Add a Comment