Authors: Chris Hoffman, Senior Product Marketing Manager
Sam Sawyer, Director of Product Marketing Embedded Storage Products
Emulex Corporation
Introduction
Several years after its initial ratification, the Serial Attached SCSI (SAS) protocol has gained momentum and is now entering the advanced stages of replacing parallel SCSI for the purpose of connecting disk drives in server applications. The majority of drives now being used for these internal storage system applications are SAS interface drives.
Within external storage systems the story is different. Fibre Channel has been the dominant protocol for providing high performance back-end connectivity from the RAID controllers through the disk enclosures and ultimately, down to the disk drives. Now with the success and efficient high-volume production of SAS drives used as internal storage for servers, the general competitiveness and even long-term availability of Fibre Channel disk drives has become uncertain. In response, storage system suppliers are rethinking the design of their external storage systems to incorporate SAS as a back-end protocol with the goal of deploying SAS drives. Like Fibre Channel, a SAS infrastructure can support the attachment of SATA drives, providing both high-performance and low-cost storage alternatives.
This article explores three alternatives for connecting high capacity, low-cost disk drives into external storage system designs utilizing the SAS protocol.
Connection Alternatives for High-Capacity Disk Drives in External Storage
Three alternative architectures are available to storage system suppliers for connecting high-capacity disk drives into enterprise-class external storage systems with SAS protocol back-ends. Each approach has various advantages and disadvantages that should be considered during architectural design. These alternatives are:
- SAS Expanders with SATA Tunneling Protocol (STP)/SATA Bridging
- Drive Tailgate Cards with Serial SCSI Protocol (SSP)/SATA Bridging
- High-Capacity SAS Drives with Pure SSP
Alternative 1: SAS Expanders with STP/SATA Bridging
Using SAS expanders with STP/SATA bridging (Figure 1) to connect high-capacity disk drives into external storage met with mixed results to date, even in small entry-level external storage systems. This option uses a low-cost Active/Active Multiplexer (A/A Mux) for high availability to convert a single-ported SATA disk drive into a dual-ported disk drive for path failover. The A/A Mux is part of a drive tailgate card located inside the drive canister with the SATA drive.
Figure 1 – SAS Expanders with STP/SATA Bridging
This approach utilizes STP with SAS expanders that have the capability to bridge the STP to the SATA protocol. One of the challenges with using STP with mixed SAS and SATA drives in the same SAS domain is that the initiators must implement and concurrently use two different protocols: STP when addressing SATA drives, and SSP when addressing SAS drives. This is a fundamental design aspect of the SAS protocol and the source of its potential promise as a low-cost method for supporting both SAS and SATA drives with the same connectivity infrastructure. In actual practice however, the use of two different protocols for disk drives in the same storage system has proven to be problematic. This is largely because the two types of disk drives, SAS and SATA, communicate in fundamentally different languages.
The complications start with the fact that host operating systems predominantly use SCSI block-level commands to access mass storage. This is effective when the target is a native SCSI device (such as a SAS drive). In the case of a SCSI target, the SCSI command is encapsulated into an SSP frame and then routed to the SAS drive. SAS drive connectivity in this alternative is fairly straightforward and works well in current deployments.
The environment becomes more complex when connecting a SATA drive using this connection alternative. When a SATA drive is the target, the SCSI initiator must first translate the SCSI command into an ATA command before it can be encapsulated into a STP frame and sent to the SATA drive. This translation process requires the implementation of a SCSI-ATA Translation (SAT) layer to convert SCSI commands into ATA commands and then convert the ATA command status responses into SCSI command status responses. As with any kind of language translation, this approach presents opportunities to misinterpret either the original command or the command response status – particularly during error recovery conditions. Unlike SCSI devices, SATA devices do not support selectively aborting pending commands. This means that during some error conditions, commands not associated with an error condition are aborted, adding complexity to the initiator implementation. If the SAT layer is implemented in a centralized entity such as the Input Output Controller (IOC) or Redundant Array of Independent Disks (RAID) controller it may introduce scaling limitations. This is due to the fact that the SAT layer must be designed to manage translation for multiple SATA drives and handle the maximum number of drives that are expected to be used in the SAS domain. Storage system suppliers often express the goal to support over 1,000 disk drives in a single storage system. A system with such a scaling limitation of using a centralized SAT layer, like Alternative 1, could have unacceptable performance while scaling up to a thousand SATA drives.
Advantages of this Alternative:
- Cost – Uses SATA disk drives which are typically the lowest cost and most widely available.
Disadvantages of this Alternative:
- Tailgate Card – A tailgate card requires space to be housed within the drive canister, which complicates mechanical design and increases the space constraints within the enclosure.
- Multiple Protocols – Requires implementation and support for two drive- related protocols, SSP for SATA drives and STP for SAS drives.
- Design Complexity – Requires the use of a SAT layer which must be managed at the initiator level.
- Not a True Multi-Initiator – An A/A Mux device is only capable of supporting one or at most two initiators, which can limit the overall performance and Reliability-Availability-Serviceability (RAS) of the storage application.
- Scalability – The SAT layer is a centralized function, which may cause unacceptable performance issues when scaling up to a thousand SATA disk drives.
- Performance – SATA devices are limited to Native Command Queuing (NCQ), which provides lower performance than Tag Command Queuing (TCQ), used with native SCSI devices.
Alternative 2: Drive Tailgate Cards with SSP/SATA Bridging
The second alternative is to use drive tailgate cards with SSP/SATA bridging (Figure 2). This is analogous to the highly effective solutions available today for attaching SATA drives into Fibre Channel storage systems. Much of that technology can be leveraged forward by SAS external storage applications as well.
This approach creates an environment similar to connection Alternative 1, except that the complexity of the STP protocol is removed. Instead, SSP is used throughout the back-end system infrastructure – regardless of the type of drive that is attached, SAS or SATA. With this option, SAS drives connect in the same manner as Alternative 1. However, for SATA drives, the bridging function is moved from the SAS expander to the drive tailgate card, which is located in the drive canister with the SATA drive. The protocol bridging is from SSP to SATA, with the SAT layer, Tag Command Queuing, Multi-initiator support and dual-port functions implemented as a part of the drive tailgate card solution.
Figure 2 – Drive Tailgate Card with SSP/SATA Bridging
Advantages of this Alternative:
- True Multi-Initiator – The SSP/SATA bridge provides a true multi-initiator implementation, as opposed to a low-cost A/A Mux device, which is only capable of supporting one, or at most two initiators.
- Tag Command Queuing – Instead of Native Command Queuing, Tag Command Queuing improves overall performance.
- Abstraction Layer – Hides the idiosyncrasies associated with different types and suppliers of SATA drives which improves the overall storage system robustness.
- Design Simplicity:
- The SAT layer is only required in the drive tailgate card, not at the initiator level.
- All protocol bridging has been pushed down to the end points (into the drive canister) allowing the use of pure SSP. Only one drive-related protocol (SSP) has to be implemented at the initiator and expander levels, avoiding potential issues with the use of SATA affiliations.
- Scalability – The SSP/SATA bridge is capable of scaling more easily than SAS expanders with STP/SATA bridging, because the SAT layer is not a centralized function. Rather, the SAT layer is implemented at the end points (in the SATA bridge chip) on each individual drive tailgate card.
- Cost – Uses SATA disk drives which are typically the lowest cost and most widely available.
Disadvantages of this Alternative:
- Tailgate Card – A tailgate card requires space to be housed within the drive canister, which complicates mechanical design and increases the space constraints within the enclosure.
Alternative 3: High Capacity SAS Drives with Pure SSP
Implementing high-capacity SAS drives with pure SSP (Figure 3) is similar to connection Alternative 2, except that a high-capacity SAS drive is used in place of a SATA drive. High-capacity SAS drives offer native SAS connectivity into the back-end of SAS external storage systems, eliminating any need for a tailgate card. To date, high-capacity SAS drives are available from only one major hard disk drive supplier.
Figure 3 – High-Capacity SAS Drives with Pure SSP
Advantages of this Alternative:
- True Multi-Initiator – The high-capacity SAS drive provides a true multi-initiator implementation, as opposed to a low cost A/A Mux device, which is only capable of supporting one, or at most two initiators.
- Tag Command Queuing – Instead of Native Command Queuing, Tag Command Queuing improves overall performance.
- Design Simplicity:
- No tailgate card is required.
- A SAT layer is not required.
- Protocol bridging is not required because only pure SSP is used.
- Scalability:
- Scaling is improved by not using a SAT layer anywhere in the system.
Disadvantages of this Alternative:
- No Abstraction Layer – Not capable of addressing potential drive vendor interoperability issues due to the lack of an abstraction layer, as is provided by the SSP/SATA bridging option.
- Cost – Capacity SAS drives are expected to have a higher cost associated with them than SATA drives since the SAS protocol interface electronics are costlier to produce.
Conclusion
There are three alternative methods available for implementing a high-capacity, lower cost tier of external storage. Each approach has its associated advantages and disadvantages. As storage system suppliers navigate the various options of new protocol implementations, it is important to optimize designs that strike the best balance between the following attributes:
- Highest level of RAS achieved
- Lowest development risk
- Best performance
- Ease of implementation
- Best time-to-market
- Lowest cost
Pure SSP approaches, as described in Figure 2 and Figure 3, are expected to be the preferred methods by large external storage system suppliers for incorporating high-capacity, lower cost disk drives into their storage system designs. These solutions provide storage system suppliers with faster time to market, increased ease of implementation, and the highest level of resiliency.
SAS is still a maturing technology, particularly in the area of using SATA drives in a SAS domain. Over the next few years, as storage system suppliers optimize their designs, we can expect to see SAS play an ever increasing role in providing end users with the high reliability, scalability, performance and capacity they require.
