Author: Sam Barnett, Business Manager, Serial Attached SCSI and Serial ATA Storage / Storage Networking Products, Storage Products Group, Maxim Integrated Products
As first generation systems based on Serial Attached SCSI (SAS) technology enter the market, leading server and storage OEMs are racing to deliver the next group of platforms based on an ever-evolving technology landscape. Unlike other semiconductor vendors, Maxim opted to enhance its popular NexSAS™ product family concurrently with the release of generation I systems. This leapfrogging approach to the deployment of SAS technology has led to the introduction of many new products with features found only in high availability, superior performing Fibre Channel systems today. Comprised of three new high-phy-count expanders, extensions in intelligent mux/de-mux devices, a new enclosure management/SAS backplane controller, and the industry’s best rate, agile signal conditioners, the NexSAS™ line delivers on the promise of high performance storage solutions for SMB (small/medium business) as well as enterprise systems.
The Maxim NexSAS™ product family is divided into four functional areas: expander technology, SAS/Serial ATA (SATA) support devices, server/enclosure/baseboard management, and signal conditioning. This article explores Generation 1.5 NexSAS™ expander technology in detail.
NexSAS™ Expander Technology
Designed with high-performance interconnect in mind; the NexSAS™ expander family takes advantage of decades of proven switching and interconnect technology development. A single functional design insures scalability from low-port offerings to the high-port-count devices needed for blade servers, storage enclosures, or switching/island SAN applications.
All Maxim expanders divide the core systems function into two separate components – the expander function and the management function. Included in the expander functional block are the Expander Connection Manager (ECM), Expander Connection Router (ECR), the Broadcast Primitive Processor (BPP), and the actual physical interfaces (PHYs) to the device (ranging in number from six to 36 depending on application need).
Rounding out the core functions is the management function block, which provides for expander management (SMP) and vendor-specific enclosure management functions (SSP/SES). Also included in this block are various peripheral interfaces needed for interaction in the overall system. These interfaces include two-wire serial, JTAG, UARTs, general purpose I/O (GPIO), and serial peripheral interfaces. The number and types of these interfaces vary with the particular expander and specific operating mode(s) employed.
Figure 1 outlines the basic functional block diagram of all expanders in the NexSAS™ family.
Figure 1: Generation 1.5 Expander Functional Diagram
General Features
All Generation 1.5 Maxim expanders share a common set of features:
- High Performance Switching Architecture – low latency, non-blocking switching matrix that provides up to 54 Gb/s of aggregate switching capacity (in large port-count expanders with linear scaling for lower port-count expander).
- Maxim Universal PHYs – self-configuring, rate adaptable PHYs supporting SAS initiators and SAS/SATA targets at both 1.5 and 3.0 Gb/s data rates; multiple per-phy pre-emphasis/de-emphasis capability, and multiple per-phy output levels build on over 20 years of Maxim technology. May be combined into SAS “wide-ports” of up to n phys wide.
- Virtual Initiator and Target Capability – SMP (Serial Management Protocol), SSP (Serial SCSI Protocol), and STP (Serial ATA Tunneling Protocol) initiator and target functions enable robust value-added services including self-configuration, self-discovery, SES-mastering, rouge drive identification, etc.
- Integrated Enclosure Management Subsystem – robust EM capabilities maximize OEM investment in legacy firmware/software/custom features and allow flexibility in system designs.
- Integrated Ethernet Interfaces – two 10/100 Ethernet MACs are included in the EM subsystem to support blade server applications or other applications where Ethernet access to the expander is required.
- Multi-affiliation STP/SATA bridge – extensions of the original STP/SATA bridge provide for two concurrent (active/active) NCQ affiliations between initiators and targets using Maxim expander technology. This innovative approach eliminates the initiator starvation issues commonly associated with the single affiliation mechanisms in SAS 1.1.
- End-to-end SAS-2 Zoning – for security, control, and performance, both SAS-2 PHY-resolved and address resolved zoning are supported.
- Numerous other enhancements beyond Generation I devices including EPOW, integrated A/D, LPC for trusted platform module interconnect, and parity protection of the on-chip memory.
Recent additions to the NexSAS™ family include the VSC7156 24-PHY, VSC7157 36-PHY, and VSC7158 18-PHY expanders. Like their predecessors, the VSC7153, VSC7154, and VSC1755 (Generation I expanders), these devices are also targeted at enterprise server, storage enclosure, and blade applications.
The feature additions outlined above bridge the gap between the SAS 1.1 and planned SAS-2 specifications and raise the bar on both functional integration and price/performance for today’s SAS market.
Two of the key differentiating features, the multi-affiliation STP/SATA Bridge and SAS-2 zoning are examined in more detail in the following sections.
Understanding the Maxim NexSAS™ Multi-Affiliation STP/SATA Bridge
STP/SATA Bridge Background
The SAS 1.1 specification defines the STP/SATA bridge as the expander component that provides the “bridging function” between STP-capable initiators and SATA targets (HDDs). Without an STP/SATA bridge, Serial Attached SCSI could not support SATA drives. In general, the SAS 1.1 specification defines the function of the STP/SATA Bridge as follows:
- In SAS expanders, the STP/SATA bridge allows the connection of SATA drives to the SAS domain
- The SAS protocol is used for connection setup/tear-down between the host (initiator) and the STP/SATA bridge
- The STP/SATA bridge only passes native SATA protocol to the SATA drove during connection
- Expanders typically have one STP/SATA bridge available in each PHY, but the bridge is enabled only when connected to a SATA device
Figure 2 graphically illustrates the STP/SATA bridge concept.
Figure 2: STP/SATA Bridge
Unlike SAS drives, SATA drives have no concept of multiple host or multi-initiator concurrent access. During a “session”, one STP initiator will maintain an “affiliation” with the SATA target. During this “connected state”, any other STP initiator attempting to gain access to the same SATA device will receive an OPEN_REJECT (STP RESOURCES BUSY) and will be forced to retry the connection attempt. This simple affiliation mechanism maintains coherency by allowing commands from only one host to be active on the drive at any given time.
When all outstanding commands from a given initiator have completed, the initiator should send a special CLOSE (CLEAR AFFILIATION) primitive to the STP/SATA Bridge, resulting in release of the affiliation and allowing other hosts to access the drive/resource.
As you can imagine, generation I expander devices support only a single affiliation in compliance with the SAS 1.1.specification.
With any new technology, implementation tends to uncover certain undesired behaviors. The original STP/SATA bridging mechanism was no different. Limitations of the originally envisioned bridge involve both fairness and performance. Manifestations of these limitations are outlined briefly below:
Fairness
- There are no formal limits imposed on the longevity of a given affiliation, meaning that an affiliation could be established forever
- Fair access to SATA devices must be enforced by the initiators and in absence of host-to-host communications, this may not be assured
- Software algorithms to achieve fairness have proven painful for OEMs
- Existing real-world STP initiators don’t play fair and occasionally fail to release affiliations
- If a valid affiliation exists between the expander STP/SATA Bridge and an initiator, other hosts cannot perform any inquiries to the drive, including diagnostic ones.
Performance
- Single threaded host access to SATA disks is slow in load balancing systems
Figure 3 depicts the SATA Affiliation concept in SAS.
Figure 3: The SATA Affiliation Concept
The NexSAS™ Generation 1.5 expander products address both the fairness and performance limitations of the SAS 1.1 STP/SATA Bridge. Through the updated bridge function, two hosts can simultaneously have active commands on the same SATA drive.
Figure 4 illustrates the Multi-affiliation STP/SATA bridge operation.
Figure 4: Multi-affiliation STP/SATA Bridge Operation
Details of the new bridge operation are summarized below:
Intelligent Connection Management
- Flexible, non-exclusive (overlaps are allowed) disconnect policies including:
Timed disconnect (multiple modes and ranges)
Disconnect after any FIS transfer
Disconnect after data FIS transfer
Disconnect after interlocked operations - Automatic callback of the proper initiator (host) based on queue tag/context
NCQ and PIO Command Support
- Supports utilization of all 32 NCQ commands tags (or fewer by configuration) on the SATA drive
- Manages interleaving of NCVQ and PIO commands from different hosts
- Tags are transparently managed and remapped to/from the drive to avoid tag number collisions between initiators (no special host software or firmware is required)
In all, the multi-affiliation STP/SATA bridge employed by all Generation 1.5 NexSAS™ expanders enhances fairness, performance, and system design convenience. Similarly, it benefits any SAS domain using SATA disk shared by two or more hosts with an architecture that can scale for future products requiring even more active affiliations while remaining completely compliant to the SAS 1.1 and proposed SAS-2 specifications.
Understanding the Basics of SAS-2 Zoning
As storage OEMs and integrators embarked on the introduction and deployment of SAS based systems, many longed for the days of Fibre Channel and some of the enterprise features it offers for traffic management (performance) and security.
To address those demands, the concept of end-to-end zoning was introduced into the proposed SAS-2 specification (please refer to www.t10.org for the latest on the zoning specification) by key industry leaders.
The SAS protocol, as defined, supports up to 16,384 devices (initiators, targets, expanders, and/or virtual devices) in a given domain; however, practical implementation of such large topologies is quite problematic. In essence, end-to-end zoning provides the mechanism necessary to manage these large physical topologies in smaller logical (segmented) groups. SAS zoning provides this function as well as the mechanisms to control access among and between groups.
SAS-2 proposes dividing specification zoning into two areas: PHY-resolved zoning and address-resolved zoning. Both approaches utilize identical access control methodologies but differ in zone group allocation.
PHY-Resolved Zoning
In a PHY-resolved zoning implementation, each PHY of “zoning capable” expander is assigned to a zone group and any device attached to that PHY (or port in the case of PHY grouping for wide ports) becomes part of that zone group. A permission table in the expander contains access control permissions that outline the protocol for “sessions” between devices in differing zone groups (by default, all devices with a zone group can interact with each other).
PHY-resolved zoning is ideal for smaller topologies, blade-server, or clustering applications.
Address-Resolved Zoning
Address-resolved zoning provides a layer of abstraction not present in PHY-resolved zoning. In an address-resolved scheme, self-discovering expanders interpret each device address and populate a table. This table contains routing information and zone permissions. Permissions in the table as well as zone group assignment are combined to determine what level of interaction any two devices may have with each other.
Typically, address-resolved zoning is used for larger topologies or where more granular control is needed.
NexSAS™ SAS Support/SATA Products
Maxim provides a wide range of SAS support (SATA) products that can be used seamlessly with the expander offering. These products range in complexity from simple 2:1 mux devices to advanced active/active solutions that improve overall connectivity, reliability, and performance in SAS servers, storage enclosures, or blades.
NexSAS™ Signal Conditioning Products
Maxim also provides complementary signal conditioning products from its storage product portfolio for those applications where long trace length or out-of-spec interconnects require signal improvement. Designed for multiple applications, these rate and protocol adaptable devices set the bar for flexibility and performance in signal conditioning today.
Conclusion
The future of SAS storage systems based on NexSAS™ technology are dependent on the needs of the end-user. Comprehensive solutions like those provided by Maxim enable unique OEM applications while diverse features of discrete components give way to multiple usage models.
With any new technology, be it SAS or something else, one size will never fit all. Proper solutions will differ by application, connectivity requirements, scalability, performance, and price sensitivity – all functions of innovation.
