Author: Mike Micheletti, Product Manager,
Lecroy Corporation
Serial Attached SCSI (SAS) is a storage interface developed to meet the needs of enterprise-class storage applications. Central to the SAS value proposition is the ability to scale storage systems beyond the limits of today’s parallel SCSI using a switching technology called Expanders. Designed to provide fan-out architecture for SAS, expanders allow SAS initiators to connect up to 16,000 physical devices in a single domain.
Storage integrators are now busy validating multi-terabyte storage subsystems that include active backplanes containing multiple expanders with 64 or 128 disks in a single enclosure. The SAS architecture is designed to seamlessly scale performance as the number of targets increases. But, as with any new technology, integrators pushing the envelope with these external RAID configurations may encounter problems not seen in simple internal direct-attached storage (DAS) applications.
Command Queuing
SAS devices support SCSI-based command queuing, which allows SAS / SATA host controllers to transmit multiple commands to a single target without waiting for the initial command to complete. HDDs that support queuing will reorganize non-sequential I/O requests to process them in the most efficient order (also referred to as scatter / gather IO)
Queue Depth
The SAS protocol includes a 16-bit queue tag which provides a theoretical queue depth of 65,000 tags (most drives maintain a practical limit of 128 tags per device). Random I/O operations (commonly associated with transaction processing) are more likely to fill this queue depth during periods of peak utilization. It’s here we find subsystems that work flawlessly at low link utilization rates and yet exhibit problems during periods of high activity. In all likelihood, these systems may be encountering time-out issues with queued commands. Queued commands that are slow to complete or fail to complete entirely have a noticeable effect on performance. These commands that are acknowledged by a device but fail to complete are surprisingly difficult to isolate:
Issues Affecting Command Time-out Problems
- Time-out problems are often intermittent, so reproducing the issue is extremely time consuming.
- Built-in recovery mechanisms in SAS and SATA protocols often mask this issue for software-based monitoring tools. In most cases, commands that time out are simply re-transmitted by the initiator, making it difficult to detect the fact that a problem occurred.
Traditional Analyzers Can’t Track Command Time-outs
- Time-out issues often emerge during periods of high bus utilization where transfer rates can range from 3.0 to 12 Gb/s. This generates an enormous number of frames and primitives that drastically reduce the amount of elapsed time that can be recorded with conventional analyzers.
- SAS supports multiple initiators (up to eight initiators in single domain) which create added complexity because they can each transmit queued commands to each device in the subsystem concurrently.
- Eight initiators x 128 tags (typical SAS HDD queue depth) x 128 HDDs = 262,144 potential outstanding queued commands
- To effectively track every outstanding command to completion would theoretically require an analyzer which was capable of maintaining individual free running clocks on over 256,000 queued operations
In some cases, when an initiator detects an incomplete, it may transmit a QUERY TASK task-management function in the next connection. (This allows the initiator to determine whether the command was received). While some protocol analyzers can trigger on task-management functions, latency in transmitting the task-management frame can allow critical details to escape from the analyzer memory.
Another issue affecting capture of this issue has been observed in both SSP and STP transactions where a specific I_T_L pair can be starved of data. This can occur when commands with the immediate bit set take precedence over queued commands. This can slow command completion times without actually causing a time out or CHECK CONDITION. The absence of hard-error occurrences makes triggering on these events with conventional analyzers difficult.
Fortunately, storage integrators now have new test strategies specifically designed to debug Command Time-out problems. Specialized tools are designed to track every command issued with an independent timer. These specialized analyzers trigger if any command is pending for more than the user-defined timeout. Traces provide a log of all commands with markers at each time-out violation. This alone can save weeks of manual analysis by identifying the exact sequence of events that preceded the time-out condition.
- Find latent performance problems – Analysis tools that provide an adjustable command threshold timer (from 50 µs to 10 seconds) allows test engineers to identify a range of issues — from fatal errors to marginal commands. By using a large time-out threshold it can help identify device “hang” conditions early in the debug process. By lowering the time-out threshold it can identify “dead-locked” connections, as well as slow, or even data-starved commands.
- View Detailed Command History – After triggering, these systems display a log for each operation with the initiator, Target ID, and other command level parameters including logical block addressing (LBA), OP Code and transfer length. Each entry includes an absolute timestamp and delta time between Command and Status. Such systems are capable of maintaining a history of the last eight million commands with markers at each time-out violation.
With large RAID subsystems, small issues can have a large effect on performance. Enterprise storage OEMs are designing sophisticated external storage subsystems with multiple initiators and dozens of SAS drives are finding first-hand that command time-out issues can materially delay “release-to-manufacturing” (RTM) for their SAS-based solutions. Increasingly sophisticated analysis tools are making the road to SAS easier for large storage integrators.
