Author: Sam Barnett, Product Line Manager, Serial Attached SCSI
and Serial ATA Storage & Storage Networking Products, Storage Products Division,
Vitesse Semiconductor
Serial Attached SCSI (SAS) offers a wealth of benefits to the enterprise server and enclosure customer (high reliability and performance, mixed enterprise/desktop drive support, improved economies of scale), and key to scalability and performance is the expander.
This article offers a brief technical exploration of SAS expander technology and its potential evolution into the “interconnect-fabric” of choice for midrange and enterprise storage systems of tomorrow.
The author assumes the reader has some knowledge of point-to-point storage architectures and SAS nomenclature. Detailed specification information on SAS can be found on the T10 website under http://www.t10.org/ (click “Drafts” then “Serial Attached SCSI (SAS)” under “Protocols and Physical Layers”). Similarly, the SCSI Trade Association (STA) also offers a robust set of tutorials on SAS. The reader may reference this material at http://www.scsita.org.
Understanding SAS
Serial Attached SCSI is the evolutionary follow-on to the parallel SCSI interface. Like other serial technologies for storage, Serial ATA (SATA) and Fibre Channel (FC), SAS was originally envisioned as a point-to-point, drive connection mechanism only, but it has become much more. In its simplest configuration, SAS provides a physical connection between a host controller and some number of targets. Figure 1 graphically illustrates this connection type.
Figure 1
As the standard gained momentum, it became apparent that OEMs needed a more robust, expanded connection construct for supporting large storage topologies. The concept of an expander was born. Like Fibre Channel switches, the expander provides a switching matrix for connection of multiple devices (host controllers (initiators), hard disk drives (targets), and other expanders) with a SAS domain. In fact, the expander’s principal responsibility is to facilitate initiator (host) communications with a target (HDD).
The SAS protocol supports very large, and sometimes complex, topologies. In a fully expanded domain, up to 16,384 devices (including expander virtual devices) can be attached via cascades and different connection routing mechanisms (direct, subtractive, and table routed). Figure 2 illustrates a large SAS topology using expanders.
Figure 2
Expander Types
There are two types of SAS expanders defined by the specification: the edge expander and the fan-out expander. Each type provides the connection mechanisms necessary to connect multiple targets to either a single or multiple hosts, one connection at a time. By definition, an edge expander can support up to 128 SAS PHYs (typically a single PHY with a physical connection point are associated). Edge expanders can be grouped together into Edge Expander Device Sets to provide for higher connection density than a single edge expander alone. In general, edge expanders are best suited to those designs where cost is a significant consideration and storage scalability requirements are limited. Figure 3 illustrates an Edge Expander Device Set.
Figure 3
Fan-out expander devices provide multiple tiers of expansion capability and are generally regarded as the more scaleable of the two types. Similarly, most fan-out expanders employ large routing tables to address downstream devices in excess of its physical connection capability.
Expander Internals
Since expanders provide a fabric-interconnect, a general fabric definition of actual function applies. Loosely defined, a fabric is a pathway on computing, networking, or storage devices that provides chip-to-chip, adapter-to-adapter, or device-to-device connections for transferring information within computing, networking, or storage systems/subsystems. In essence, the expander is a switch or cooperative switching facility.
Regardless of the underlying architecture employed, a SAS expander consists of several key elements: the Expander Connection Manager (ECM), Expander Connection Router (ECR), Broadcast Primitive Processor (BPP), as well as device interfaces, which vary with actual implementation.
Figure 4 graphically illustrates the expander device and Figure 5 illustrates the expander device model.
Figure 4: Expander Device
Figure 5: Expander Device Model
Expander Connection Manager
- Maps a destination SAS address in a connection request to a destination PHY using direct, subtractive, or table routing
- Arbitrates and assigns or denies path resources for connection requests following SAS rules for arbitration and pathway recovery
- Configures the “Connection Router”
Expander Connection Router
- Routes signals between pairs of PHYs (initiators and targets) as configured by the “Connection Manager”
- Provides routing resources needed to support connections
Broadcast Primitive Processor
- Routes “topology” messages to appropriate devices (e.g. topology changes)
Device Interfaces
- Provide physical layer device interfaces to internal and external components
- Arbitrate and route frames between PHYs
- Provide physical and link layer connections to auxiliary I/O (e.g. 10/100 Ethernet, GPIO, etc.)
Interaction of the subsystem (block level) components making up the expander device is rather complex but is clearly defined in the SAS specification. A detailed analysis of expander operation is not covered here.
Deployment Models
There are two major models for attaching SAS HDDs and targets to the host. The simplest model allows for direct connection to the host via a host controller without expansion (See Figure 6).
Figure 6
In this approach, scalability and physical connection capabilities are strictly limited to the number of PHY ports on the controller. Similarly, no aggregation of bandwidth is performed in the direct-connect model, meaning that much of the inherent performance provided in the SAS controller goes unused.
The second approach employs an expanded architecture that allows for a more flexible and scalable design that can take advantage of the bandwidth available through the SAS host controller. The expanded model provides for significant flexibility in both server storage and storage subsystem architectures (See Figure 7).
Figure 7
SAS Expanded Architectures in Scalable Storage Subsystems (JBODs)
Large storage subsystems are primarily designed through the use of high-port-count expanders. As mentioned, these devices are capable of addressing a large number of dual-port SAS drives (targets). Systems employing these architectures provide an array of robust fault tolerance alternatives depending on the specifics of the implementation. In most cases, external connections are provided to allow for storage subsystem scalability. As the storage system grows, the root expanders provide the addressing capability needed to connect all devices down stream. [Note that each dual-port drive requires an address (World Wide Name) essentially requiring two SAS addresses per physical drive.] Figure 8 graphically depicts a simplistic subsystem with failover capability.
Figure 8
Economical Deployment Models
Expanded architectures within a DAS deployment model allow for storage scalability and have the ability to address two distinct economic models with the same design. An enterprise-level system is deployed by the selection of SAS HDDs with the end user receiving all the expected performance and reliability provided by similar parallel SCSI systems today, but with the advantages of SAS. Similarly, a second economic model is achieved by utilizing the same design and using SATA HDDs.
Future Directions
As with any technology, its future will be determined by the needs of the ultimate consumer of devices and systems built on the technology. Speed grade increases and additional features will be most prevalent within the SAS community itself. However, as chipsets become more and more integrated, the way storage systems are designed today may change. It is reasonable to assume large chipset vendors will adopt a universal controller architecture, meaning that systems deploying their technology can take advantage of any storage technology by simply attaching the expansion facility and drive of choice to their interface.
Summary and Conclusion
In all, the future of storage and storage networking is contingent upon the evolution of new protocols and architectures – like SAS. Expander technology provides the scalability and connection flexibility needed to address different market segments and economic cost points. Even so, the proper solution for a given challenge will differ by application, connectivity requirements, scalability, performance, and price sensitivity.
What does this mean for SAS’ future? Only time will tell.
