STB Suite

Upgrade to 12G HDD/SSD Testing

12 G HBA Results – Backwards

Summary

Yes, here’s the summary first.

In case you’re not interested in the why’s. But if you are you’ll find them at the bottom of this article

The Bottom Line at the top

More drives per day out the door.
That’s it – you can run more drives at a time, at higher speeds per drive.

And that translates into more drives processed per day.

And that number goes right to your bottom line.

Yes, there is an investment in new hardware, mostly in enclosures. Possibly some in a new test computer with PCIe-3 slots. A 12G HBA is about $400.

But the ROI is good and fast.

A “power system” (PC with PCIe-3 slots, 12G HBA, 12G enclosure/cables) is going to net you six or seven times more drives out the door per day.

Our example test system (Corei5, 16GB, PCIe-3 slots) cost about $500 to build from scratch.

Width is the key

Think of the maximum bandwidth or throughput as lanes on a highway.

You may not have a car that drive at 1,000 mph. But you could have 10 cars each driving at 100 mph.

More lanes (bandwidth) means more cars (disks) at a time can drive (test) at full speed.

You may be testing older drives, 3G SAS and SATA drives.

But you will still see a vast improvement in drives-out-the-door.

That’s the bottom line – you can run more drives at a time, at higher speeds per drive.

And that translates into more drives processed per day.

And that number will quickly work its way to your bottom line.

Now for some details

History and Future Roadmap

The speed or throughput history of SAS and SATA devices has grown or increased dramatically. Beginning at 1.5Gbps, scaling up to 3G, 6G, and now 12G (SAS drives only) per SAS port, with 24G (again, SAS only) on the horizon. The SATA specification at this time does not extend beyond 6G.

What do the speeds really mean? Translation…Drilling down to reality

It’s really fairly simple. A SAS port which can transfer data at 12Gbps has a maximum throughput of just that – 12 GigaBITS per second.

Gb vs GB vs MB

The little “b” means bits, but the real world of working with disk drives is used to numbers expressed in Bytes, not bits.  A little later on we will convert our G bits to  G Bytes.

 

Number of Ports

A typical 12G SAS HBA has 8 SAS ports. There are also HBA’s available with 16 ports.
More ports = wider highway = more cars.

PCIe Bus Speeds

But how fast can your test computer transfer that data from the SAS interface into or out of computer memory?

That speed is a factor of what type of PCIe bus you are plugging your card into.

So – you must consider PCIe “type” and “width” – so, another table to consider, including another 20% “lost” to 8b/10b encoding.

Typical SAS HBA’s will be 8-lane PCIe type of card.

Table 3 PCIe transfer rates

PCIe Type Data Rate Total bandwidth – 8 lanes Minus 8b/10b Overhead
PCIe-1.1 250 MB/s 4 GB/s 3.2 GB/s
PCIe-2.0 500 MB/s 8 GB/s 6.4 GB/s
PCIe-3.0 1000 MB/s 16 GB/s 12.8 GB/s

As you can see, your test system’s PCIe bus speed is just as important as your HBA speed.

A commitment to move up to 12G may involve a new test PC, as well as 12G HBA(s), cables, and enclosures.

 

A handy utility to show you what type of PCIe bus(s) your test system has is called

HWinfo  and can be downloaded here – http://www.hwinfo.com/download.php

Here is a screenshot of one of our test systems that’s about 4 years old

Note: the LSI SAS HBA (LSI-9200-8e) is plugged into a PCIe-1.1 slot. This will severely limit the actual throughput.

In actual fact this test setup can run at about 900MB/s max. In other words, with average disk drives (150 MB/s) we can only test 6 drives at a time at full speed. After six everything starts to slow down.

Bottom line – you won’t ship very many drives per month with this setup.

Here’s a newer system – the ~$500 example, with a $400 LSI9800-8e HBA

Ah – much better! This setup should be able to run at least 40 of the same 150 MB/s drives at a time, at full speed.

There is an up-front cost for new hardware (test system, HBA, cables, enclosures) – but with a 7-fold increase in product output those costs will be paid back in no time.

 

Enclosures and cables must be up to snuff

Signal integrity at 12G speeds is super-critical. For everything to work at the maximum speeds you must have 12G rated enclosures using expanders and cables.

Adding it all together – what’s the most you can expect?

One more table to compare 12G & 6G SAS and PCIe 3, 2, & 1 – best to worst, taking into account the maximum enclosure speed.

Assuming that each enclosure with expander will use 4 SAS ports

These numbers are the maximum you should expect

PCI, HBA type & enclosure
type
Enclosure/Cables # of full speed (120 MB/s) drives
PCIe-3, 12G HBA 12G 40
PCIe-2, 12G HBA 12G 25
     
PCIe-3, 12G HBA 6G 18
PCIe-2, 12G HBA 6G 18
     
PCIe-3, 6G HBA 6G 18
PCIe-2, 6G HBA 6G 18
PCIe-1, 6G HBA 6G 12
     
PCIe-3, 6G HBA, 3G enc 3G 6
PCIe-2, 6G HBA, 3G enc. 3G 6
PCIe-1, 6G HBA, 3G enc 3G 6

So – a new system with PCIe-3 and 12G SAS hardware will net almost 6 times more drives out the door per day.

 

Important things to be aware of

You can’t skimp on cables or enclosures. Even if you are not using enclosures with expanders you still must maintain high quality in the entire signal path.

You can check your signal integrity using the STB Suite (of course!). Simply run some typical tests, then go into STB Original mode. Right-click on a SAS HDD/SSD and choose View Log Pages. Look at Log Page 0x18 – here’s an example of what bad cabling looks like –

What you should see is zeros.

 

Want to see if your drives are running at their full rated speed? Right-click again and choose “Edit Mode Pages”. Select Page 0x19 SAS SSP page, then click “Edit Page”
Look at what the Negotiated Link Rate and compare it with the Programmed and Hardware Physical link rates.

Don’t scrimp

Depending on how good or bad you cable situation is you might end up with 12G drives only negotiating to 6G speeds. And then having intermittent errors and retries.

The lesson is this – don’t try to save money on cables.

It is definitely not worth it.

In summary, the summary

For upgrading your test systems to 12G we recommend our friends at PC-Pitstop.

Ask for Paul, he will guide you to the best fit for your particular needs.