Author: Cameron T. Brett, Manager, Product Marketing
PMC-Sierra, Inc.
Putting Serial Attached SCSI (SAS) to work is relatively easy. Server and storage solutions can be created ranging from simple direct-attached storage (DAS) in servers to high-availability, switched rack environments. To look at these solutions, it is best to understand the SAS building blocks first.
SAS Components
Controllers – Host Interface
Controllers create the connection between SAS disk drives and the host system, typically a server or an external RAID controller. The host connections used today are PCI-X or PCI-e with the controller embedded on the motherboard or mounted on an add-in card.
Expanders and Switches
Expanders are essential to SAS systems by allowing a single SAS link (commonly four x1 links combined into a single wide port) to connect to many SAS ports ranging from 12, 18, 24 or 36 ports, creating very large storage configurations. The expander internally handles all of the routing and contentions for link usage with the added benefit of a management interface that allows for statistics gathering and fault isolation. Expanders are typically found on backplanes or add-in cards.
Figure 1: Expanders Connect a Single SAS Link to Many SAS Links
Expanders can be cascaded (connecting expanders to expanders) to provide connectivity to an even greater number of devices. Expanders can connect to multiple hosts as well as to other expanders. In this case, the expander acts as a switch, similar to those in an Ethernet or Fibre Channel environment.
Figure 2: Cascading Expanders Allow for Large Configurations
Disk Drives
SAS disk drives are central components in a SAS environment, as are Serial ATA (SATA) disks. With SAS support for SATA disk drives, many configurations are possible, accommodating high-capacity, performance, or tiered storage. SAS drive capabilities include 15K RPM, 300GB capacity and 3 or 6Gb/s data transfer rates. SATA drives are available up to 10K RPM, 1TB in capacity and 3Gb/s transfer rates.
SATA Port Multipliers
SATA disk drives used in a high-availability, failover system must be dual-port. A port multiplier allows a SATA disk to be connected to two SAS controllers, such as a SAS I/O or RAID card. If connectivity to one of the controllers fails, the other controller can continue to read/write to the SATA disk.
Creating SAS Solutions
SAS is a very flexible technology that enables a wide range of server and storage solutions. The applications for SAS include simple DAS in a server, high-capacity tiered storage, high-availability shared host systems, and use as a switched fabric in a rackmount environment.
Simple SAS Configuration
SAS is a replacement for Ultra320 SCSI. Initial SAS applications will replace future designs that might have used Ultra320 SCSI. For example, a server would use an embedded SAS controller and/or a SAS host bus adapter (HBA). Inside the box, the controller is inserted into a PCI-X or PCI-e slot. The HBA has two mini SAS 4i internal connectors (with red cables) which connect to the internal disk drives. Each mini SAS 4i cable/connector could connect up to four disks. The SAS/SATA drive connectors include both a single SAS port and power in the same connector. The cables are routed to the boot drives. Internally, the x1 connector is used on both SAS and SATA drives because internal server applications typically do not require the use of high availability connections.
High Capacity (8TB) Tiered Storage Solution
One step beyond using SAS for primary storage in a server, SAS could be used for nearline storage. In this implementation, a standard rackmount server with an internal SAS HBA, would attach to eight 1TB SATA disks for a total storage capacity of 8TB. This design preserves the legacy SCSI boot model and brings a significant value proposition into the mainstream server market. It also enables applications requiring high-capacity storage in a server. SAS drives could be added to serve a high-performance database, online transaction processing or intensive web applications.
Scaling Outside the Server
SAS can scale outside the box beyond traditional DAS limitations. This is accomplished by taking the embedded SAS controller or HBA in the server and attaching it to a SAS switch or external enclosure.
Figure 3: Connecting to JBODs Outside the Server
Cascading Enclosures
A feature of SAS cabling called the Universal Connection, enables cascading of enclosures. With SAS, it is important to understand the downstream and upstream connections to ease configuration of cascaded JBOD/RAID subsystems. This is accomplished by using two different connectors; one that is keyed for only an IN or OUT connection and one that is keyed for either IN or OUT connection. In some instances, the application is sensitive to the cascading of enclosures, therefore this scheme was devised to ease configuration. In any case, the enclosure manufacturers will provide the appropriate labeling to avoid improper cascading.
No matter how the cascade cabling is done, the end device will always have a universal connector.
Figure 4: SAS Connection Symbols
Shared Host Implementation
Larger configurations can be created with shared or dedicated storage on multiple servers. One method of doing this is with a SAS switch, to connect up to nine distinct wide (x4) links connected to multiple servers and multiple JBOD enclosures. This can be taken to the next level by cascading multiple JBODs together. Because of the significant bandwidth of a wide link (12Gb/s half-duplex and 24Gb/s full-duplex at 3Gb/s), up to three or four JBOD enclosures can be cascaded onto a single wide link accommodating hundreds of drives. This SAS expander application will also simplify backup of one system to another, fail-over, system upgrades, simultaneous use of production systems and test systems all through the same set of hardware.
Figure 5: Multiple Servers Sharing JBOD Storage via SAS Switch
Scalable, Sharable DAS = SAS as a Fabric
The next table compares the differences between SAS and other available technologies, including Fibre Channel and Gigabit Ethernet. Fibre Channel is a performance- and distance-oriented technology, with one major disadvantage – it is relatively costly. Conversely, Gigabit Ethernet is low cost and can go very long distances, but it is limited on performance compared to these other solutions. 10 Gigabit Ethernet is very good on performance and distance, but its cost is relatively high. SAS offers a great blend of excellent performance combined with a cost lower than other technologies can offer. While SAS does not offer a long distance option, the distances in today’s standard rackmount configurations can easily be covered to accommodate most servers and storage applications. Powerful systems can be configured using this method. SCSI Trade Association (STA) believed this market was under-served by the other technologies such as Fibre Channel or Gigabit Ethernet. For single-room configurations, SAS is the most cost-effective and highest performance alternative.
Figure 6: SAS as a Fabric Fills a Need for High-performance with Low Cost
In-rack Switched Environments
This configuration demonstrates a rack-mount system that includes two servers, a switch and four JBOD enclosures. There are 52 disk drives – not counting the eight in the second server – that are shared by the two servers. Each server has access to a large number of disks with a very manageable cabling infrastructure. The cables are small and flexible and take up little space as well as insuring a simplified configuration.
High-availability SAS in External Storage
Since SAS drives are dual-port, SAS enables high-availability external storage applications. In an external enclosure there can be a large number of drives and depending on the drive carrier, it may accommodate SAS or SATA drives. If using a SATA drive, a port selector dongle is required to enable redundant availability to the drive, which is typically built right into the drive carrier itself. In high-availability applications, the enclosure would have redundant power supplies and cooling systems in place, along with redundant external controllers. These external controllers could be Fibre Channel, SAS or even Gigabit Ethernet which would enable an iSCSI design. Most likely these controllers would include RAID functionality to preclude any single point of failure.
Figure 7: SAS in a High-availability External System
Summary
SAS is a robust, flexible technology that works in a broad variety of systems whether simple server DAS, external storage environments, or high-availability, fault-tolerant systems. With the use of expanders, a low cost, high-performance fabric can easily be created for multiple-server/multiple-storage connectivity in a rack or blade environments. The old chapters on parallel SCSI have given way to SAS — and the many new capabilities it delivers. Stay tuned, there are more SAS chapters to come.
