Trends in Ethernet Storage Fabric

One of the few things constant and inevitable in the world and our lives, is change. Adapting to change in meaningful ways can bring better outcomes and a brighter future. But to control change, the IT world relies on methodology and standards. Our challenge in the IT space is to integrate cost-effective solutions that enable a business, keeping it more agile while transforming in a highly competitive ever-changing landscape.

In the past “de facto” guidelines, such as the ITIL framework lead to more consistent IT best practice for Service Managemen. However, best practice is not good enough for the science of electronic communication where standards allow competition to flourish when the largest companies are pre-dominant.

Interconnects and their Standards body

For Super-computing, HPC and high-end database, InfiniBand(IB) was the interconnection of choice, for high-speed, low latency and reliable interconnection of servers, with its standards governed, and the technology promoted by the InfiniBand Trade Association (IBTA). As an interconnect, IB competes with Ethernet, Fibre Channel, and Intel Omni-Path.

The American National Standards Institute (ANSI) oversees a sub-committee T11 INCITS which has driven the Fibre Channel (FC) ANSI standards since 1994. It is storage over fiber only. They used the spelling “fibre” to differentiate the standard, different than the multi-mode fiber used for connectivity. FC was designed as a serial interface to overcome the limitation of SCSI and to provide in-order and lossless delivery of raw block data. It established 1Gb in 1997 and has doubled every few years since.

The Institute of Electrical and Electronics Engineers (IEEE) Standards Association (IEEESA) is an organization within IEEE that develops global standards for a broad range of industries. IEEE Standards Association 802.x helps with standards for Interworking/Security 802.1, Wireless LAN 802.1x, and specifically IEEE 802.3 which governs Ethernet standards for the IT marketplace.

Scale-out HCI VM Solutions and their associated Storage and Software Defined Storage using Ethernet is the focus of this blog. Ethernet Storage Fabric is the move towards lower latency, higher performance top of rack ethernet based switching and the server (node) NIC where standards have gone from 1Gb to 100Gb, with 200Gb QSFP56 NIC and Ethernet ports available this year. This is less about Ethernet Core and Campus Fabric L3 features and more about lower-cost local L2 East-West traffic and high-speed ports servicing scale-out solutions in the cloud and popular Ethernet scale-out solutions available to enterprise IT as a result.

All Ethernet Storage Fabric (ESF) solutions have a minimum of 2-4 nodes and scale their compute and storage performance as you add nodes. HCI VMware vSAN, Nutanix, SimpliVity and software-defined storage Qumulo, Isilon, Scality, Weka.io as examples. They commonly use higher performance flash caching for a lower-speed capacity tier and may leverage erasure coding or higher number of nodes for redundancy, reliability and performance.

A Look Back at Ethernet Networks

Ethernet began with the dawn of the Internet in 1973. Bob Metcalfe, Dave Boggs at Xerox Palo Alto Research Center setup an early form called Arpanet funded by Defense Advanced Research Projects Agency (DARPA). On May 22, 1973, Metcalfe further described the architecture and gave it a name: Ether Network. The name stuck. Xerox proposed it to the Institute of Electrical and Electronics Engineers, and it became an open standard in 1983.

By the late 1980s, Ethernet was winning market share as the de facto standard over IBM’s Token Ring. Initially 10Mbps over thick coax cable, then standards for 10Mbps over twisted pair (RJ45) set in 1990; 100Mbps, and finally gigabit (1Gb) during the dot-com boom era. In 2002 came 10Gb (10GBASE-SR) over fiber. Similarly, 10GBase-T or 10Gb using low cost unshielded twisted pair (UTP copper) and 2007 standards for backplane ethernet over printed circuit boards (1Gb and 10Gb) fueled integration and lower cost options.

The Internet and the Public Cloud are fundamentally tied to Ethernet. With the advent of continually lower costs for server-side flash caching of capacity tier, and faster node-to-node server communication brought performance and reliability.

The Public Cloud has reset IT expectations. Hyper-Converged Infrastructure and ethernet Software-defined Storage came to market through web-scale use of commodity servers with local storage. These solutions had similar architecture to those used by web-scale cloud service providers: Amazon, Google, Facebook and others. The Internet, Public Cloud and related service providers continue to be a driving force for Ethernet Storage Fabric with and eye on open standards and lower-cost components and switching.

Due to consumer push, most are familiar with IEEE Wireless Local Area Networks (WLAN) standards which are 802.11 followed by a letter, the latest is 802.11ax. Distinct from this is the Ethernet IEEE Standards classified under 802.3. For IEEE 802.3, since the year 2000, added two letters, e.g. 802.3ab to 802.3az, 802.3ba to 802.3bz, etc. In 2010, the IEEE “802.3ba” standard for 40 and 100Gb followed, with more companies starting consortiums which make proposals to the IEEE Standards Association about backward compatibility, and physical transport.

Ethernet Changes in the Last Decade

Let’s look back to understand some key turning points in the last decade.

Fibre Channel 8Gb, 16Gb, 32Gb will have 64Gb core switching this year, and has always had very reliable, fast, lossless transport for shared SAN block storage. For Ethernet to more effectively compete it needed more reliability.

Mellanox had been the predominant force behind the InfiniBand (IB) market and InfiniBand Trade Association (IBTA), absorbing Voltaire IB Switching/40Gb Ethernet cross-over in 2011. InfiniBand relies on Remote Direct Memory Access (RDMA), the lifeblood of the world's faster supercomputers and high-end database.

Mellanox was instrumental in bringing RDMA over Converged Ethernet (ROCE) to the Ethernet market in 2011 after it saw the 10Gb Ethernet emerge and grow. ROCE v2 also brought lower latency and a move towards lossless UDP packet transmission. Though there is a segment using iWARP (internet wide area RDMA protocol) over more reliable TCP, the majority of Ethernet switching providers moved to offer ROCE due to the lower latency.

IEEE 802.3by was the ratified standard for SFP28 (10/25Gb) in June 2016 adding to earlier 2015 802.3bm standard for related QSFP28 (100/40Gb), with plans for 200Gb and 400Gb. Further acceptance and lower costs were possible since 2-strand LC OM3/OM4 fiber was already in use for 10Gb and the Fibre Channel SAN storage. Bi-Directional 40Gb or Bi-Di 100Gb transceivers could extend to 100m with 2-strand LC OM4.

SFP28 provides for 25 Gbps but is also backward compatible with 10Gb cabling. Switch ports with QSFP28 100Gb also could support 4x25Gbps (SFP28), and still be backward compatible with pre-existing 40Gb QSFP+ to allow 4x10Gb SFP+. Once standards for 100Gb QSFP28 and 25Gb SFP28 Ethernet Storage Fabric (ESF) were ratified, competition and innovation for NIC and top of rack switching were close behind. By the Fall of 2016 most server manufacturers began to OEM SFP28 NICs from Broadcom, Mellanox, Intel and Marvell (Cavium/QLogic) and offered them at a now slight premium over 10Gb. Mellanox did well with its popular Spectrum switch family offering 100Gb ports quickly, while Broadcom created ASICs Trident and Tomahawk based on the new standards for most Ethernet switch makers, Cisco included. Cisco began manufacturing its own Cloud Scale ASIC, which is the basis for its more price competitive Nexus 9300 Series.

Today newer switch models have more QSFP28 ports for uplink, switch pair interconnect and could flexibly support storage solutions which could take advantage of 100/40 and could also breakout to support HCI solutions which had 10/25Gb NICs. Most vendors competing and lowering costs for the newer models, which are now meant to supersede prior generation 40Gb QSFP and 10Gb SFP+ models. Deprecated QSFP/SFP+ will see low pricing for some time due to transition and market demand. 10G-BaseT will see growth as well due to low-cost twisted pair which is predominant in SMB where 10/1Gb share non-fiber, low priced RJ45 Cat6a/Cat7 durable PVC wired media workstations, client desktop, and lower cost consumer mass-market.

How cost-effective are some of the newer SFP28 QSFP28 Top of Rack Switches?

Surprisingly, the Mellanox Spectrum family of Ethernet switching, which HPE also OEMs as its StoreFabric M-Series, offers SN2010M 18xSFP28, 4QSFP28 which each List under $10K, 2in1U, and not much more for half-width HPE 8 port entry licensed SN2100M QSFP28 100Gb with up to 16x QSFP28 2in1U. Each QSFP28 100Gb port breaks out to 4x10/25Gbps for 56 SFP28 ports for the SN2100. HPE M-Series is the recommended iSCSI switching for 3PAR, Nimble (dHCI) and Primera, and is now incorporated into HCI/SDS solutions for the Nth Labs Experience showcase. Nutanix had early alliance with Mellanox and uses their 10/25 SFP28 NICs. Mellanox designed a powerful network orchestration provisioning product (NEO) for the AHV hypervisor, which of course is supported by the Mellanox Spectrum HPE M-Series as well with Nutanix on ProLiant/Apollo DX-Series.

Extreme Networks acquired ROCE models from Brocade VDX and offers the new standard switching, including entry VSP7400 for HCI/SDS. We like them for Campus Fabric and innovative SPB. HPE Aruba similarly blending the standards for campus Ethernet, tied to best of breed Wireless with ClearPass security. HPE Synergy used the new QSFP28 100Gb as the basis for its latest Virtual Connect VC100 F32 ports. Synergy Compute are 2x25 or 2x50Gb for ESF iSCSI and lowering the cost of converged FCoE 32Gb “on demand” native FC licensing to SAN fabric as well as dedicated 32Gb FC interconnects. Synergy compute support vSAN HCI with internal 40-drive D3940 and local chassis caching of PCIe NVMe Optane 2.5in. So more traditional density solutions, as well as HCI and SDS benefit from the advent of lower cost, but higher speed Ethernet Storage Fabric.

What is coming Next?

Ethernet IEEE working groups provided a clear standard for higher speeds as well, up to 400Gb. NIC vendors are already delivering dual port 100Gb NICs, and with ProLiant Gen 10 Plus PCIe 4.0 new QSFP56+ 200Gbps NIC options are available. With Server based 3D NAND flash, NVM Express (NVMe), NVMe Optane (6x less latency than NVMe), and now Intel Optane DC Persistent Memory, the solutions were needing the higher speed transport now available. Ethernet Switches with QSFP56 200Gb (4x50Gb), or QSFP28 100Gb backward compatible ports, are due from several vendors in 2020, with 400Gb on the horizon.

Gartner stated in 2017, a major Fabric inflection point is coming with NVMe over Fabric (NVMe-oF). Fibre Channel is ahead with already ratified NVMe-oF hba and FC SAN Switching, with Ethernet due to have ratified standards by the end of 2021. Eventually, with end to end compliance, native OS driver support, the NVMe-oF protocol will replace the lower speed and higher latency of the existing iSCSI protocol for improved flash and NVMe storage performance. Some ESF switch vendors already marketing switches as NVMe-oF ready.

Have you found the right mix of Hybrid IT for your business model?

As Ethernet Storage Fabric became faster and more reliable, the market demand for scale out Software defined storage, backup and hyper converged Ethernet based storage solutions increased. The move to simpler, cost-effective pay as you grow on-premise hybrid ethernet based solutions has been driving lower cost multi-cloud enterprise solution designs with increased TCO and ROI benefits versus Public Cloud.

Nutanix, VMware vSAN, SimpliVity, Cohesity, Qumulo and Synergy solutions, all available via OEM on HPE Gen10 iLO5 Root of Trust platforms, helping to protect from firmware tampering or compromise of the component supply chain. Nth Generation has these solutions and more available as part of your Nth Labs Experience.

Today, our world is still dealing with COVID-19. The IT industry is constantly evolving and changing, with more affordable solution purchase options such as HPE GreenLake, a cloud consumption model, or when needed 0% financing, 90% payment in 2021 continue to provide relief.

Nth Generation is not just our family, it's yours, and we want to help you prove IT is more crucial now than ever in making the core business more agile. We pride ourselves in being your trusted advisor. Call Nth Generation to let us help you find the right mix. We are here to help! Jim Westover Enterprise Solution Architect