When it comes to design, making things elaborate and complicated is not necessarily the best way to go. Keeping things simple while focusing on delivering outcomes that match the customer expectations (if not exceeding them) is what matters. I was once again in beautiful Silicon Valley in November 2017 at Storage Field Day 14 where E8 Storage introduced us to their shared NVMe storage solution.
The elegance, simplicity and feeling of raw power emanating from their solution reminded me of the awesome Vision Mercedes-Maybach 6 Cabriolet: a timeless car blending 30’s streamliner style and influences from Riva wooden yachts with today’s advanced technology and a beast of an electric motor. Let’s hop into this marvel and ride along the scenic road of a blog post to discover E8 Storage.
Company Overview
E8 Storage is a storage startup founded in 2014 by Zivan Ori and Alex Friedman. Per the traditional Crunchbase review, E8 Storage have levied cca 18M USD in 2 rounds of financing. The company is headquartered in Santa Clara, CA, in the USA.
Both co-founders worked previously at XIV during the IBM era, holding management positions in the XIV Research & Development team. Other core members brought in their expertise and embed the DNA from past companies where they worked at, such as Envara, Wanova, Primary Data, NetApp, Nexenta and others.
Architecture
E8 Storage is a distributed storage solution that entirely leverages NVMe Flash memory. It consists of a storage node (the current models are the E8-D24 and the E8-S24, more on this later) that combines two x86 servers and 24 NVMe drives. More below!
Software
E8 Storage is, in a certain way, a distributed shared-nothing solution (but can also be configured to share larger LUNs between hosts) where each host I/O interactions with the storage appliance are direct and do not reach other hosts. Traffic is only of north-south type (between hosts and the appliance). No east-west traffic is seen as there are no controller VMs residing on the hosts so there’s no communication between nodes as you can see in HCI systems. The approach taken was to use a PCIe-to-Ethernet bridge and an aggregator to present the NVMe storage to each of the hosts. The aggregator component is in fact made up oftwo x86 servers within the enclosure that map the NVMe drives and expose them via a some of the 100 GbE ports allocated to each of the x86 servers. E8 software runs on these x86 boxes, acting as a metadata server and presents services such as APIs, GUI, etc.
To connect to the storage nodes, an agent software (which acts in fact as a storage controller) must be installed on each host that needs access to the storage – this software acts as a single controller that serves I/O from and to the host. A single appliance can serve up to 96 hosts. In terms of resource consumption on the hosts, only 1 physical CPU core is needed per host (as only a single agent instance runs on a host).
From a data protection standpoint, multiple data protection schemes are available, however E8 recommends RAID6 with 16+2 stripes. Because the data is striped in chunks and not at the drive level, there is no waste of disk space. The system is also able to handle upgrades in a non-disruptive way. Hold on for performance and the architecture that enables, we’ll cover this in a dedicated section below.
Hardware
From a hardware perspective, there are two models available: The E8-D24 (for Write-Intensive applications) and the E8-S24 (for Read-Intensive applications). From a performance perspective the same latency is available on both appliances, but performance varies based on the model (more on the “Performance” section below). E8 Storage systems use Dual ported Intel DC D3700 (E8-D24) and DC D3600 (E8-S24) NVMe drives based. I think I’ve heard that solution leverages the Intel SPDK toolkit, about which I wrote approximately a year ago (but struggling to find sources on that, so happy to be corrected). At the time of writing, each appliance fits 24 NVMe 2.5″ SSD drives within a 2U enclosure.
From a connectivity perspective, appliances sport up to 8x 100 GbE ports. Hosts need to have RDMA enabled NICs to connect with E8 appliances. Those can be either 10 GbE or go up to the 25/50/100 GbE speeds. Undoubtedly however customers who look into this level of performance and throughput/latency should consider bandwidth usage to avoid network throughput from becoming a potential bottleneck. And as usual, it’s better to check with the vendor as specs may change.
The way HA is implemented in E8 is specific, as it’s the first NVMe HA enclosure with dual swappable controllers. The use of dual ported NVMe drives ensure each drive is visible from each controller and besides that the enclosure has all of the usual high availability features we’re used to in the x86 world i.e. dual PSUs and a passive mid-plane design. Finally, a battery backup unit is present as well.
I could run on and on, but for the sake of brevity I’ll leave things here and invite you to view the video of the technology deep-dive with co-founder Zivan Ori at Storage Field Day 14:
Performance
From a performance perspective, E8 Storage will just blow your mind away. E8 storage indicates read latencies of 100 μs and write latencies of 40 μs using 4 KB blocks with a queue depth of 1 (QD=1) for both appliance models.
Mind Blown by the IOPS, throughput and latency – The Singularity is near!
From an IOPS and throughput perspective (and using the parameters above), the E8-D24 (Write-Intensive) reaches 10M Read IOPS @ 40 GB/s throughput and 1M Write IOPS @ 20 GB/s, while the E8-S24 (Read-Intensive) reaches 5M Read IOPS @ 20 GB/s and 200K Write IOPS @ 8 GB/s. The throughput differences are due to the amount of ethernet ports (the E8-D24 leverages 8x 100 GbE ports while the E8-S24 leverages 4x 100 GbE) and the write IOPS difference on the E8-S24 are due to the Intel DC D3600 NVMe drive who does only 30K IOPS/drive (compared to the DC D3700 which does 100K IOPS/drive). How is such a performance possible?
E8 Storage have explored the classical architectures that we’re accustomed to see in the storage world and found out that none of these would suit their requirements to achieve the desired performance and squeeze out every drop of performance from each single NVMe drive.
Without going into much detail (at the risk of transforming this piece into an essay or a short novel) the short story is that the architecture is designed to be massively parallel and involves patented technologies, as well as a separation between the control path (where the servers running the E8 agent connect to the metadata servers on the E8 appliance to get information about reads or writes) and the data path (actual reads or writes to the NVMe drives). For the data path part, E8 maps 1 queue from each NVMe drive to each host. Because each NVMe drive has 65535 queues (as opposed to SAS/SATA which have only 2 queues) this allows for massive parallelism without prioritization of I/O or wait times.
E8 – Data Path vs Control Path (screenshot from the video stream)
There’s obviously much more to say about it, but if you want to dig out more please do watch the videos recorded during E8 Storage presentation at Storage Field Day 14.
Market Orientation
The current market orientation, as of Q4-2017, is clearly within Tier-0 applications requiring high throughput, high IOPS and low latency. Verticals that may see such requirements are in the financial services especially high-frequency trading, but also any other industry or application where low latency and high data ingest/processing speeds are critical to achieve primary business outcomes and obtain a competitive edge over their industry contenders.
E8 Storage is well designed to cope with multiple simultaneous streams of data or applications that massively parallelise their I/O operations. In the next 18 to 36 months however this solution could also accommodate Tier-1 workloads. I expect Flash to slowly commoditise while demand for Flash keeps growing not only for Tier-0 but also for Tier-1 applications as more speed is requested by application owners and applications alike, while the demand for CPU and RAM also increases at increased paces. To be honest with the reader, I haven’t noticed that demand is falling or that applications are being dramatically optimised/re-architected to reduce their footprint 😉
One point to take in consideration is that we could see in the future (and I’m purely throwing an hypothesis) that the current line with NVMe drives in E8 Storage could see the addition of another line of more powerful appliances using Dual ported Optane NVMe drives which offer even lower latency and far superior write throughput, on the condition that such drives are produced in the adequate form factor, are also dual ported and will have sufficient capacities to support the E8 architecture. And obviously, that it makes sense for E8 Storage to leverage these. It turns out that on their site they have an “Optane Ready” statement on their Solutions page so maybe I’m not far away from the next big thing with E8.
We didn’t get access to pricing, though Co-Founder Zivan Ori mentioned that while similar in price to other AFA solutions from a capacity perspective, their solution is designed to be much faster, with higher IOPS and dramatically lower latency than current mainstream AFA offerings. Definitely worth looking for all of us who are looming into getting an AFA solution, or who are looking for the best performance possible at an affordable price point. Then, we have to get into what “affordable” stands for. But if we make again car parallels, you could get the performance of a Bugatti Chiron for the price of a Lamborghini Huracan (and without the expensive off-sides these cars usually command).
Max’s Opinion
E8 Storage was for me the most exciting presentation at Storage Field Day 14 (closely followed by the magistral VMAX presentation by Vince Westin, that I hope to cover soon). It’s not every day that you see a radically new approach to storage and in their case they definitely have a set of firsts to their record. I like that they introduced us not only to the technology, but to the thought process that led them where they are and how they deconstructed the common approaches to AFA architectures and came up with a magnificent architecture, technology and product.
Beyond that, the E8 Storage team did put a lot of brain power around maximising the use of capabilities offered by NVMe drives and by RoCE (RDMA over Converged Ethernet); and bind these altogether with their own patented technologies in a solution that delivers astounding performance at the same price point (per E8) of current NVMe-enabled AFAs. Remember, we’re talking here about latencies >=100 μs, 40 Gbps throughput, 10M read IOPS and 1M write IOPS for an E8-D24.
As I close this post, I’d like to return back to the Mercedes-Maybach 6 metaphor. Unlike its car counterpart, E8 Storage is not just a feat of engineering and/or design well done, it’s a production ready solution that is likely to address the needs of the most demanding workloads in the market it intends to target. And that’s the beauty of a well-engineered solution: addressing real-world requirements with an effective and elegant architecture. And believe me or not, just like some enterprises are desperately after the fastest storage system in the market, I wouldn’t mind to have a Mercedes-Maybach 6 address my “transportation requirements”.
Watch these guys, they may be up to great feats.
Disclosure
This post is a part of my SFD14 post series. I am invited to the Storage Field Day 14 event and Commvault GO by Gestalt IT. Gestalt IT will cover expenses related to the events travel, accommodation and food during the Commvault GO and SFD14 event duration. I will cover my own accommodation costs outside of the days when the events take place. I will not receive any compensation for participation in this event, and I am also not obliged to blog or produce any kind of content. Any tweets, blog articles or any other form of content I may produce are the exclusive product of my interest in technology and my will to share information with my peers. I will commit to share only my own point of view and analysis of the products and technologies I will be seeing/listening about during this event.