In January 2019 I attended Cisco Live alongside with Tech Field Day Extra. During that event, we had a presentation on the latest generation of Cisco’s HCI solution, Cisco HyperFlex (HX). The outcome of this presentation was a somewhat passionate blog post I wrote on the matter, where I pointed some doubts and questions (if not outright outrage) regarding the NVMe-based HX performance, as well as their optional hardware acceleration card.
I had the opportunity to discuss with the Cisco HyperFlex product team following my article on the latest release. Cisco wanted to clarify some of the concerns that I had regarding their latest HyperFlex improvements, namely their additional hardware acceleration card, and also the NVMe architecture performance.
Because this turned out to be a very constructive discussion, I decided to publish a follow up post covering the clarifications, as well as amending parts of my previous article. I will also provide some comments in my usual “Max’s Opinion” section.
NVMe Architecture Performance
Designing an NVMe architecture with Reliability, Availability and Serviceability (RAS) in mind has one challenge. The PCIe bus to which the NVMe drives are attached to is directly connected to the CPU. This means that there is no disk management interface such as SATA/SAS or SCSI to handle events such as drive failures or hot-plug / hot-remove events.
The consequence is that without any sort of management layer that can control events and take the hit, any of these events happening on a PCIe bus will throw an exception that is likely to fully crash the system. Since storage systems need to be fully redundant, and a kernel crash is a catastrophic failure that will likely impact data consistency, a solution has to be found.
For this purpose, Intel developed the VMD driver (Intel Volume Management Device), which can be used in systems with Skylake processors. This element is key to support RAS requirements by providing an abstraction layer that handles such unplanned events in place of a hardware disk management interface.
In the context of Cisco HyperFlex, the move to an all NVMe environment means that a lot of work had to happen under the hood (lot of quality assurance work & testing). Without the necessary stability on handling these events, the move to NVMe drives would be a hazardous one.
For their initial platform release on NVMe, Cisco has understandably set the priority on stability, i.e. ensuring all potential exceptions are accurately handled by the Intel VMD driver. This priority given to stability means that performance gains will be handled on a second stage.
From a performance perspective, the HyperFlex product management team stated that past the initial release, they are already seeing performance improvements with the current development work that is in progress. These performance improvements should keep improving as the development teams work on the product code and on leveraging further optimisations. These future improvements will be delivered to HyperFlex NVMe customers via software updates.
Cisco was also keen to share more details on the NVMe drive types and capacity specifications:
- Cache drives: 3D XPoint (Intel Optane) 375 GB with 60 DWPD (Drive Writes Per Day)
- Capacity drives: 3D NAND (unspecified manufacturer) 1TB and 4TB drives (total of 8 capacity drives per node)
Hardware Acceleration Card
Cisco have also provided additional insights on their hardware acceleration card, the HyperFlex Acceleration Engine (HAE). This card is a newly introduced optional device that is compatible only with the new HyperFlex models.
On that matter, Cisco stated that while the HX software handling inline deduplication and compression already provides very high performance, the hardware accelerator offers even further gains: higher compression rates and higher performance. And of course CPU offloading.
Cisco also stated that while the HyperFlex architecture guarantees deterministic latencies and avoid two phase deduplication and compression, the hardware offloading adds higher compression ratios and performance to this deterministic latency.
What is relevant above is that the introduction of the HAE isn’t negatively impacting existing customers, who will still benefit from software improvements made to the HX software. HAE is nevertheless available to those with higher requirements in terms of deduplication and compression gains.
There is a lot more things that were presented in the January’s Tech Field Day Cisco HX presentation at Cisco Live Europe. Topics such as stretched clusters, container capabilities, as well as Cisco Intersight, HyperFlex Edge and more were covered.
I recommend to watch the relevant Tech Field Day video to catch up on these and see where the HX platform has moved to.
This has been an interesting article to write, especially in the context of my previous February 2019 article, which has been moderately annotated to reflect the changes induced by the one you are reading now.
Not that it would be a full mea culpa, but sometimes we end up writing only with the information that is available to us. And doing that while being a bit too passionate (or explosive, you call it) can lead to incorrect information being communicated. In that sense, I am glad the Cisco HyperFlex product management team took the time to read my article and to arrange a briefing to address my questions.
This raises an important point: when presenting at public events, it’s worth addressing to the right audience. I love Tech Field Day events, and TFDx at Cisco Live Europe is always a bit special: I’m the only non-networking guy among a panel of networking semi-gods. Usually, we delegates get to see several Cisco teams presenting on sometimes totally unrelated topics.
As the only Cisco storage product, the session on HyperFlex is usually the pinnacle of the TFDx / Cisco Live experience for me. And while writing this article (but also when talking with the HX product management team) it came to my mind as whether this almost incendiary blog post I wrote wasn’t all about improper messaging directed to the wrong audience?
It’s difficult to let the messaging about an infrastructure / storage product hit home to a network experts crew. Choices need to be made to simplify the messaging. When you have one lone wolf storage guy in the panel, and messaging is simplified, what can possibly go wrong? Perhaps it wasn’t even Cisco’s fault, after all, they were not addressing to a storage analysts audience, and I just happened to be at the wrong place and at the wrong moment.
That is where I think that Cisco needs to present their HyperFlex product to the right audience. Tech Field Day Extra is nice for starters, but I would definitely love to see Cisco present during a full-fledged Storage Field Day. It’s really where the momentum on storage products is happening, and where the dialogue can take place between the product management crew and the delegates.
This leads to the topic of putting more emphasis on design challenges. Putting more emphasis on the challenges introduced by using NVMe drives on a PCIe bus is something that would have certainly been a great discussion to have at Storage Field Day. Understandably, it’s really hard to spend time on this in a less than 40 minute presentation that was meant for a more generalist audience.
Talking about the solution itself, I find myself having much more confidence on the NVMe architecture part and the great explanations provided by the product management teams have dissipated my concerns. I am still however neutral on the HAE card. I’m not really a big fan of custom cards, but if someone finds its happiness there, why not.
This post is a part of my TFD Extra at Cisco Live Europe 2019 post series. I am invited to the event by Gestalt IT. Gestalt IT will cover expenses related to the events travel, accommodation and food during the event duration. I will not receive any compensation for participation in this event, and I am also not obliged to blog or produce any kind of content. Any tweets, blog articles or any other form of content I may produce are the exclusive product of my interest in technology and my will to share information with my peers. I will commit to share only my own point of view and analysis of the products and technologies I will be seeing/listening about during this event.