Author: Mike Micheletti
Computer Access Technology Corp. (CATC)
Scalability can be defined as “the ease with which a solution or component can be modified to fit the problem as the size of the problem increases”. Scalability in the storage domain allows IT managers to respond to changing storage requirements within the enterprise, and is a stated design goal of Serial Attached SCSI (SAS).
The SAS protocol delivers scalability via Expanders, integrated circuits (IC) that function as switch components and route frames between initiators and end-points. By providing multiple point-to-point drive connections in a single IC, expanders allow aggregation of bandwidth between Initiator and multiple SAS disk drives. Expanders also enable an architecture that accommodates dynamic discovery and configuration as new components are added to the system.
In the SAS topology there are two types of expanders: Fan-out and Edge. The server-attached storage market will typically use edge expanders, which can address up to 128 SAS addresses or drives in a segment. When a fan-out expander is incorporated into the architecture, up to 128 segments can exist within a SAS domain, which allows SAS to address up to 16,384 SAS physical links (PHY). Compared to the parallel SCSI limit of 15 devices, SAS storage topologies can support large, enterprise-class applications.
To facilitate cost-effective manufacturing, most early expander components typically have 12 or 16 PHYs within a single ASIC. These components are targeted at rackmount or server storage applications allowing from six to 12 target devices in a storage shelf. (Remember, the initiator will normally use two or four of the expander ports). Applications requiring a higher number of SAS end devices may group several edge expander devices together on a single printed circuit board (PCB) known as an Edge Expander Device Set (see figure A).
Edge expanders maintain a direct routing table that identifies all SAS addresses that are directly attached to the expander. In addition, SAS expanders may use table or subtractive routing to direct frames to PHYs that are more than one level removed from the expander.
As shown in figure A, the links between the initiator and the expanders use four PHYs combined into a “wide” port to provide higher bandwidth link between SAS end devices. The lower edge expanders use the direct routing method to send frames to the SAS disk devices that are directly attached. The topmost edge expander would use an internal table of addresses to route frames to the attached devices, known as Table Routing.
Figure A: Typical edge expander device set configuration with all 12 ports on each expander fully populated
For larger configurations, two edge expander device sets may be attached using their Subtractive Routing Ports. The subtractive routing port is used for frames addressed to devices that are not found in the expander’s direct or table route list. When sending frames out the subtractive routing port, the expander is effectively saying “I don’t recognize this address, you handle it…” Without the subtractive routing method, SAS expanders would be required to maintain large table route indexes (up to 16K addresses) that would be cumbersome to configure and search (see figure B).
Device Discovery
By definition, the ability to easily add capacity to an installed SAS topology is an important attribute of a scalable architecture. The SAS protocol meets this challenge by automatically detecting and enumerating the SAS components with minimal user intervention. SAS utilizes an innovative approach to automatically discover addresses for all the SAS components and configure the routing tables for use by the protocol. SAS begins this process at power-on by sending Identify Frame. SAS uses serial management protocol (SMP) commands to configure the expander topology:
Report General tells the initiator how many indexes are in its route table, the type of routing and how many PHYs are attached.
Report Mfg Info Request provides information on the expander vendor ID and product revision. It’s generally used for implementing vendor-unique features within the system.
Discover Request is the SMP command that queries a device to determine what’s attached to each expander PHY and the associated SAS address. Among other things, the discover response allows the system to determine if Serial ATA tunneled protocol (STP) or serial SCSI protocol (SSP) should be used to communicate with attached devices.
Report SATA PHY is sent to determine the SAS address for any SATA devices that are reported by the discovery response.
Configure Route Information Request builds the SAS table entries for expanders that have configurable route tables.
Report Route Information Request is transmitted to gather the route information (if the Report General Response indicates the expander is self-configuring).
SAS expanders with their built-in ability to detect and configure pathways to all attached devices allow users to create storage systems that are both scalable and reliable.
