12 G HBA Results – Backwards
Yes, here’s the summary first.
In case you’re not interested in the why’s. But if you are you’ll find them at the bottom of this article
The Bottom Line at the top
More drives per day out the door.
That’s it – you can run more drives at a time, at higher speeds per drive.
And that translates into more drives processed per day.
And that number goes right to your bottom line.
Yes, there is an investment in new hardware, mostly in enclosures. Possibly some in a new test computer with PCIe-3 slots. A 12G HBA is about $400.
But the ROI is good and fast.
A “power system” (PC with PCIe-3 slots, 12G HBA, 12G enclosure/cables) is going to net you six or seven times more drives out the door per day.
Our example test system (Corei5, 16GB, PCIe-3 slots) cost about $500 to build from scratch.
Width is the key
Think of the maximum bandwidth or throughput as lanes on a highway.
You may not have a car that drive at 1,000 mph. But you could have 10 cars each driving at 100 mph.
More lanes (bandwidth) means more cars (disks) at a time can drive (test) at full speed.
You may be testing older drives, 3G SAS and SATA drives.
But you will still see a vast improvement in drives-out-the-door.
That’s the bottom line – you can run more drives at a time, at higher speeds per drive.
And that translates into more drives processed per day.
And that number will quickly work its way to your bottom line.
Now for some details
History and Future Roadmap
The speed or throughput history of SAS and SATA devices has grown or increased dramatically. Beginning at 1.5Gbps, scaling up to 3G, 6G, and now 12G (SAS drives only) per SAS port, with 24G (again, SAS only) on the horizon. The SATA specification at this time does not extend beyond 6G.
What do the speeds really mean? Translation…Drilling down to reality
It’s really fairly simple. A SAS port which can transfer data at 12Gbps has a maximum throughput of just that – 12 GigaBITS per second.
Gb vs GB vs MB
The little “b” means bits, but the real world of working with disk drives is used to numbers expressed in Bytes, not bits. A little later on we will convert our G bits to G Bytes.
Number of Ports
A typical 12G SAS HBA has 8 SAS ports. There are also HBA’s available with 16 ports.
More ports = wider highway = more cars.
PCIe Bus Speeds
But how fast can your test computer transfer that data from the SAS interface into or out of computer memory?
That speed is a factor of what type of PCIe bus you are plugging your card into.
So – you must consider PCIe “type” and “width” – so, another table to consider, including another 20% “lost” to 8b/10b encoding.
Typical SAS HBA’s will be 8-lane PCIe type of card.
Table 3 PCIe transfer rates
|PCIe Type||Data Rate||Total bandwidth – 8 lanes||Minus 8b/10b Overhead|
|PCIe-1.1||250 MB/s||4 GB/s||3.2 GB/s|
|PCIe-2.0||500 MB/s||8 GB/s||6.4 GB/s|
|PCIe-3.0||1000 MB/s||16 GB/s||12.8 GB/s|
As you can see, your test system’s PCIe bus speed is just as important as your HBA speed.
A commitment to move up to 12G may involve a new test PC, as well as 12G HBA(s), cables, and enclosures.
A handy utility to show you what type of PCIe bus(s) your test system has is called
HWinfo and can be downloaded here – http://www.hwinfo.com/download.php
Here is a screenshot of one of our test systems that’s about 4 years old
Note: the LSI SAS HBA (LSI-9200-8e) is plugged into a PCIe-1.1 slot. This will severely limit the actual throughput.
In actual fact this test setup can run at about 900MB/s max. In other words, with average disk drives (150 MB/s) we can only test 6 drives at a time at full speed. After six everything starts to slow down.
Bottom line – you won’t ship very many drives per month with this setup.
Here’s a newer system – the ~$500 example, with a $400 LSI9800-8e HBA
Ah – much better! This setup should be able to run at least 40 of the same 150 MB/s drives at a time, at full speed.
There is an up-front cost for new hardware (test system, HBA, cables, enclosures) – but with a 7-fold increase in product output those costs will be paid back in no time.
Enclosures and cables must be up to snuff
Signal integrity at 12G speeds is super-critical. For everything to work at the maximum speeds you must have 12G rated enclosures using expanders and cables.
Adding it all together – what’s the most you can expect?
One more table to compare 12G & 6G SAS and PCIe 3, 2, & 1 – best to worst, taking into account the maximum enclosure speed.
Assuming that each enclosure with expander will use 4 SAS ports
These numbers are the maximum you should expect
PCI, HBA type & enclosure
|Enclosure/Cables||# of full speed (120 MB/s) drives|
|PCIe-3, 12G HBA||12G||40|
|PCIe-2, 12G HBA||12G||25|
|PCIe-3, 12G HBA||6G||18|
|PCIe-2, 12G HBA||6G||18|
|PCIe-3, 6G HBA||6G||18|
|PCIe-2, 6G HBA||6G||18|
|PCIe-1, 6G HBA||6G||12|
|PCIe-3, 6G HBA, 3G enc||3G||6|
|PCIe-2, 6G HBA, 3G enc.||3G||6|
|PCIe-1, 6G HBA, 3G enc||3G||6|
So – a new system with PCIe-3 and 12G SAS hardware will net almost 6 times more drives out the door per day.
Important things to be aware of
You can’t skimp on cables or enclosures. Even if you are not using enclosures with expanders you still must maintain high quality in the entire signal path.
You can check your signal integrity using the STB Suite (of course!). Simply run some typical tests, then go into STB Original mode. Right-click on a SAS HDD/SSD and choose View Log Pages. Look at Log Page 0x18 – here’s an example of what bad cabling looks like –
What you should see is zeros.
Want to see if your drives are running at their full rated speed? Right-click again and choose “Edit Mode Pages”. Select Page 0x19 SAS SSP page, then click “Edit Page”
Look at what the Negotiated Link Rate and compare it with the Programmed and Hardware Physical link rates.
Depending on how good or bad you cable situation is you might end up with 12G drives only negotiating to 6G speeds. And then having intermittent errors and retries.
The lesson is this – don’t try to save money on cables.
It is definitely not worth it.
In summary, the summary
For upgrading your test systems to 12G we recommend our friends at PC-Pitstop.
Ask for Paul, he will guide you to the best fit for your particular needs.