This post is part of the blog series related to Storage Field Day 11. Find out the entire SFD11 content, presentations, articles, presenting companies and delegates here.
During Storage Field Day 11, Intel presented us their Storage Performance Development Kit (SPDK). What does this stands for and what are the advantages? And most of all, why should storage generalists and end users care?
A Tale of Latency
In the beginning was the HDD. And the HDD spinneth slow. And then came the SSD. And with SSD came speed. But speed was still subject to the limitations of the legacy SATA or SAS bus interfaces.
In SAS 2.0 and SATA III, the theoretical maximum transfer rate of 6 Gbps mainly addressed the way to interact, send commands and store/retrieve data on spinning drives. Even while the first SSD drives were whoopingly fast (by the standard of the time), using for example the SATA bus interface with AHCI meant a single command queue (and a shallow depth with only 32 commands per queue). This meant that no matter how fast an SSD drive could theoretically be, there was still latency either at the drive level (for entry-level models) or at the controller level.
With the introduction of NVMe, a protocol dedicated to high performance flash media that leverages the PCI Express bus, an unparalelled breakthrough was achieved (pun intended) in paralellization of I/O, with 65535 queues and 65535 commands per queue. In addition to this, new generations of flash media are faster and faster.
This breakthrough in performance and reduction of media latency means that software latency, which was largely overlooked in the past due to its residual fraction in the overall latency stack, is now becoming a pain point that needs to be addressed, as you can see in the chart below.
Media Latency: The Challenge – graphic courtesy of Intel
Intel SPDK – An Introduction
Intel developed their Storage Performance Development Kit and the NVMe Polled-Mode Driver as a response to this shift in latency towards the software stack. Note that this SPDK implementation currently supports a few Linux implementations only.
Normal drivers work in kernel space with context switching and interrupts. Interrupts by their very definition cause the CPU to “interrupt”, i.e. to stop processing a task and immediately execute the cause of the interrupt. Likely, context switching means that the CPU must switch the context of an operation it was doing, i.e. park aside the current work, process something else and resume the previous operation later. With the extremely low latencies new flash devices are attaining, operations in kernel space are becoming costly in terms of time and cause bottlenecks and a solution must be sought to eliminate or reduce these.
What SPDK does is changing the game in how drivers work: Intel NVMe Polled-Mode Driver lives in user space instead of kernel space. Because kernel space is not touched, this allows for smoother operation because less CPU cycles are wasted than would with interrupts and context switching.
While this picture depicts Intel Data Plane Development Kit, it is still relevant to understand how the Polled-Mode Driver lives in user space and where storage developers can place their custom code.
Another major change is the modus operandi of Polled-Mode Drivers vs normal drivers. Intel have a great analogy on their website: think of performing operations with normal drivers as trying to hail a taxi cab on a Friday evening. Many taxis are passing by, but most are already carrying someone, leading to increased waiting times. Compare polled-mode drivers with the taxi queue at a major international airport. Taxi cabs are ready and waiting for passengers, loading them as they arrive and the next taxi picks another passenger. The same goes for operations performed through Polled-Mode Drivers. Operations are immediately dispatched as the driver is constantly listening and waiting for operation requests.
This frictionless method of operation allows to operate at μsec levels of latency instead of milliseconds previously, thus effectively fighting back the issue of software bottlenecks, especially at the driver level.
SPDK and Software-Defined Storage
Intel provided an overview of what their polled-mode SPDK driver operating in user space can achieve vs regular NVMe drivers operating in kernel space. The results below are breathtaking and the slide speaks for itself, even if this is a sample workbench.
Max’s Opinion
Why should SPDK matter to us mere mortals and storage consumers, and why should we care about this complicated stuff? We should care because of the massive impact that SPDK may have on new generation storage solutions.
I think that SPDK holds immense potential for SDS projects; SDS is likely here to stay and it’s in software space that we are seeing most of today’s storage innovations. At CPU core parity, SPDK addresses much more I/O than with a regular kernel space driver. This means that an SDS project leveraging SPDK can expect to achieve much more IOPS than they would normally with the same CPU consumption. Or eventually reduce their CPU requirements.
SDS projects will be able to focus their R&D on building the core DNA for their solution (think caching, deduplication, compression, erasure coding, encryption etc.) instead of working on driver-related aspects. Likely, they can expect SPDK-based solutions to provide more CPU cycles to execute their app DNA instead of making difficult balancing choices between kernel-space driver operations and their SDS solution fundamentals.
With SPDK, Intel enables the storage industry (or at least the part of the industry leveraging Intel flash) to leverage the full potential of NVMe flash technology in a way that was impossible before.
Software-defined storage solutions will greatly benefit from this potent ally and we can expect to see in the future even more powerful solutions where latency is further decreased, performance increased, and freed-up CPU cycles used to further improve data storage, protection, availability, etc. At the sad price, maybe, of hearing some contenders brag about their new megazillions of IOPS.
Related posts by Storage Field Day 11 delegates
Resources
- Introduction to the Storage Performance Development Kit (SPDK)
- New Storage Performance Development Kit goes live
- Storage Performance Development Kit Community
- Accelerating your NVMe Drives with SPDK
- 01.org / Intel Open Source / Storage Performance Development Kit
Disclosure
SFD11 Disclosure: this post is a part of my SFD11 post series. I am invited to the Storage Field Day 11 event by Gestalt IT. Gestalt IT will cover travel, accommodation and food during the SFD10 event duration. I will not receive any compensation for participation in this event, and I am also not obliged to blog or produce any kind of content. Any tweets, blog articles or any other form of content I may produce are the exclusive product of my interest in technology and my will to share information with my peers. I will commit to share only my own point of view and analysis of the products and technologies I will be seeing/listening about during this event.