SATA Brings RAID5 Home
by Mark Foster
February 2004 (updated 8/04)

I was faced with a storage problem. I had a need to address the increasing demands for storage space on my primary home system, but I also needed to deal with resiliency and cost issues. My growth in demand is to support in-home video service. We use ReplayTV to record material and DVArchive to store that MPEG2 content on a PC. The DVArchive PC can then serve the content back to the ReplayTV player. At the lowest quality setting, it’s about 1GB/hour of content. Higher quality and fast changing scenes can easily quadruple that number. I was running out of space on my DVArchive server and needed a cost effective solution to increasing it’s capacity.

The easiest thing to do would have been to add another IDE disk, now reaching about 300 GB. I’d added a 170 GB disk about a year ago, and it was nearly full. I could replace the full disk with a 300 GB drive, moving all the content to it, and that would probably last me about another 9-12 months. But that wouldn’t address the risk I faced in having all the content on a single disk and no (reasonable) means to back it up. Although most of the current disks are designed for 3-5 years, with the manufacturers typically offering a one year warranty, design longevity is getting a little more dicey. Besides, even if I can get a failed disk replaced, how do I get back the 300 GB that was on it? I needed a solution that provided the ability to grow, and be able to easily withstand a disk failure.

For backup, I could have added two 300 GB disks, and created a mirror (RAID 0) configuration. There are a number of inexpensive IDE or SATA raid controllers that would support a two-disk mirror. But this would have left me with little or no capability of expansion, unless I setup a second two-disk configuration. I didn’t like the thought of getting less than 50 cents on the dollar for my disk storage, and it also seemed that as the storage demands approached the 1 TB mark, I’d be looking at an awful lot of disks – more space, more noise, more heat dissipation. This is my home, not a datacenter. I gained considerable work experience with RAID5 SCSI arrays when they were first becoming popular many years ago. With the introduction of high capacity, low-cost SATA drives, and some promising reviews of SATA RAID controllers, I decided that this might be both an effective approach to solving my storage problem and a fun project.

I did look at the handful of low cost RAID5 bundled solutions that are available. They all seemed fairly expensive to me (in total cost and cost per MB). Part of the problem is most of those systems include a fairly beefy standalone processor; some include an additional hardware RAID controller. I already had a decent host system (2.4 GHz P4). I didn’t need to have dual-host access to the array, nor some of the other features in the pre-built bundles. I wanted to be able to increase my array incrementally, both as my needs increased and as disk capacities increased. Besides the cost drawbacks, I wasn’t confident the bundled systems would be “open” enough to permit me to grow in the way I wanted. So I embarked upon the details of the key piece: a RAID5 controller that could support several low cost disks.

Tom’s Hardware provided a very helpful article during my investigation. For those without much background in the concepts, they provide basic information on RAID configurations. They then review a handful of controllers with specific disk models, and compare the performance with some SCSI RAID controllers.

I found some other reviews helpful in my selections:
Promise FastTrak S150 SX4
Raidcore RC4852 8-port SATA Raid controller
PCI impact on RAID performance: review.

I ultimately selected the RAIDCore 4852 controller. This controller seemed to offer quite good performance characteristics. It also had some features that I thought I would find very useful: the ability to incrementally upgrade, to convert or transform an array from one type to another, and to configure a distributed spare disk (more on some of these capabilities later). [Aug 04 Update: for those that have heard of difficulties with the RC4852, they have been resolved with a replacement controller, the BC4852; the support folks at RaidCore really stepped up to resolve the problems, and provided no-cost replacements. Kudos to them.]

The next step was to look at drives. I expected that with many disks together, heat dissipation and noise were going to be two significant issues. I also wanted drives for which the manufacturer was willing to offer warranties longer than one year. Finally, the performance of the drives had to at least not create a bottleneck for the controller. Pretty much all of them succeed on this last point – my 32-bit PCI bus is the limiting factor. The drives needed to be as quiet as possible because I planned to locate the array in living space. Build up of heat is an enemy of disks, so ones that produced less heat required less demanding cooling, and thus, could use quieter fans for cooling. Less waste heat also has the nice side effect of better energy efficiency. I boiled my choice down to the Samsung SpinPoint SP1614C (160 GB, 7200 RPM SATA). At the time, these best addressed my requirements, while also offering lower cost per GB than nearly any other capacity; I got these disks for $115/ea from newegg.com.

Finally, I needed to examine my options for a hot-swap chassis (you don’t expect me to have to shut everything down and tear it apart to replace a failed disk, when all the other components can handle online removal and insertion, do you?). The RC4852 supports up to 8 SATA disks. My initial plan was to start with 3 disks, and be able to add one disk at a time, up to the capacity of the controller. There are a few SATA hot-swap cages available; most of them are designed to fit into two or three 5.25” half-height device bays. My PC’s chassis was out of the question: not enough space to add any hot-swap cage. This meant I would have to have some external cabinet to hold a hot-swap chassis. It also meant I could choose from the variety of chassis that were available. SuperMicro’s CSE-M35T offered the highest density, with 5 SATA disks in the space of three 5.25” slots, and it had a single 92mm fan that looked easy to replace, if needed, with a quieter model. My hunt for a cabinet to hold the CSE-M35T led me to HSC Electronic Supply, a local new and used parts place, where I found a 4 (or, with minor work, 5) slot cabinet with 250W power supply. It’s very similar to the Venus enclosures offered by AMS.

Given that I wasn’t going to have the disks in the same cabinet as the CPU and the SATA cables needed to go from the RC4852 to the CSE-M35T, I figured I needed to have the longest cables the SATA spec allows: 1 meter. Both the RC4852 and CSE-M35T came with an appropriate complement of SATA cables; but they were all too short: 18 inches or less. Good to have short cables if you’re staying in the same cabinet; bad if you’re not. There were surprisingly few places carrying the 1m SATA cables at the time I was shopping for them. No doubt, as SATA becomes more popular, and more folks do as I’m describing here, they’ll be in stock everywhere.


Putting it all together – The Hardware

The initial core components include three 160 GB Samsung SpinPoint disks, the RAIDCore RC4852 controller, and a bundle of SATA cables.


The RC4852 supports the extended 64-bit PCI-X bus, but works fine in less expensive motherboards with 32-bit PCI bus. Five of the SATA connectors are on the front, three on the back side of the board. RAIDCore has chosen to give hardware assist just where it’s needed, and puts most of their logic in the software that runs on the host CPU. This design means that as you upgrade your CPU and motherboard, you automatically upgrade the performance of your RAID subsystem. It also means they should be able to more readily enhance some of the subsystem’s capabilities, since doing so generally won’t mean new hardware.


The SuperMicro CSE-M35T provides all the hardware and cables to install in 5.25” drive bays and extend disk activity signals to LEDs.


It was quick and simple to install the disks in the quick-release brackets that are integral to the CSE-M35T. The SATA disks, like SCA SCSI disks, provide power and data connections in a hot-plug capable design. There is no cable or adapter between the chassis bracket and the disk: the disk connects directly to the chassis backplane.


A look inside the back cover of the CSE-M35T shows dual power connections, the SATA cable connectors, and header pins for disk activity LEDs. Also note the large rectangular holes in the printed circuit board. These are important: it’s how the air is drawn off the drives and exhausted toward the back of the cabinet.


With the back cover and fan re-installed, you can still see the SATA and power connections. The fan shown is Sanyo Denki 109P0912H202: a 55 CFM fan that runs at 2900 rpm and produces a whopping 36 dbA. I cannot blame SuperMicro for choosing this fan. It will definitely keep five drives running cool. They also probably intend their chassis to spend most of it’s time in a noisy computer room setting, not a quiet home office. I replaced the Sanyo Denki with a SilenX SX-092-14. This fan only pushes 36 CFM, but also is a nearly inaudible 14 dbA. I replaced the generic fan that came in the salvaged cabinet with a SilenX 80mm thermistor controlled fan – it will get noisy when it’s warm in the cabinet, so should function as a early warning that I need to do something different.


The cabinet was salvaged from some previous use, so had an ID switch and SCSI-2 connectors, all of which were removed for this application. I figured that even if I had to get a new power supply, the $7 I spent was a great deal on the sheet metal frame (ultimately, everything worked just fine - no replacement needed ).






To reduce the unruly mess that all the thin, but stiff SATA cables would create, I made a 5-cable bundle (I’ll make another 3-cable bundle when the time comes to add a three drive chassis and disks). If you’re doing this, be sure you line up the cable end keying, as shown in this closeup of the end of the bundle. Also, take care if you are using tie-wraps, as I did. They should be just snug, not tight. Otherwise, you risk affecting signal integrity (RAIDCore customer support confirms this). An alternative to tie-wraps is electrical tape.


One end of the bundle connects to the backplane in the CSE-M35T


The other end of the bundle connects to the RC4852, installed in the PC


The back of the assembled cabinet shows the cable bundle passing through one of the open slots previously used for a SCSI cable connector. I tie-wrapped the bundle to the cabinet at this opening, as a strain relief.

I thought about using the 3ware SATA-to-firewire cable adapters, so I didn’t have to have a single hard-to-disconnect cable set chaining the PC and external cabinet together. Between not having enough free PCI blanks in my PC and the additional problems the extra cabling and connectors could cause me (not to mention the added expense), I quickly extinguished the thought.

With the disks installed, the cabinet looks like it was made for this. Leaving the (missing) blank open will help some with airflow, and in this case, function is more important to me than cosmetics.




Activating the Array – The Software

Installing the software from the CD went without a hitch. The RC4852 comes with a comprehensive Users Guide (in PDF). Although I was impatient, it was worth reading through it. They offer two interfaces for array configuration, one BIOS level, the other a GUI that runs under Windows. If you were setting up an array that was going to comprise the system boot disk, you’d need to do the configuration from the BIOS interface. Since I was just adding an auxiliary drive, I was able to perform all the steps from the GUI. I setup the RAID5 configuration to span across all three disks. I had originally thought I’d be able to use their “spanning spare” technology, but it turns out I’d need a fourth disk for that. As easy as it is to get replacement disks, I think I can risk running without a hot spare or spanned spare for now.

Once setup, I told it to initialize the three disks and create the array. Be prepared for the time this process takes. It’s analogous to a low-level format (even though that’s not what it’s doing), and scales according to the size and number of disks. It took about 8 hours on my setup. Once that completed, I decided to run a “check consistency” (sometimes when working with fresh bits and bolts, I tend to be paranoid). That took about 2.5 hours. The check is in progress in this screenshot:


Then, I had Windows format the “disk”, the result is a single 295 GB drive G:. Recall that in RAID5, one disk worth of capacity is allocated to parity (even though the parity is striped across all the disks). So, three 160 GB disks is really only 320 GB of space, and with some filesystem overhead, 295 GB looks about right.


Time to exercise some of the features. I copied several GB’s worth of content to the new filesystem, and began playback of the video. To simulate a genuine disk failure, I pulled the second disk (“disk 01” in the array nomenclature). A popup box notified of the status change, and a glance at the management GUI confirmed the condition.


All the while, the video continued playing with no glitches or hiccups. So far, so good. Now, simulating the “repair”.

I had thought that I’d be able to reinsert the disk, tell the controller to “rescan” the drives, and I’d be quickly back in full operation. I assumed this because I’d been careful not to write anything to the disk during this failure. This simplistic view turned out to be false. That’s okay, it’s just that recovery took longer than I thought it might in this instance.

A reinsert of the disk generated a popup box.


As instructed, I performed the rescan. The system sees the disk, and recognizes that it is part of an array: note that Disk 01 shows zero space available.

But, the array is still in critical state, and there is no active rebuild or recovery task working toward fixing that state.

I proceeded to reinitialize Disk 01. This wipes out all the array information on the disk, and thus, makes it look like an unused disk. Doing so also changed the list of disks that the array knew about: Disk 01 is now listed as “missing”.


I would have thought I could then request a “rebuild” of the array, but either the sequence of events I had followed, or the support for that (direct) approach isn’t fully there. In any case, I decided to try the “transform” feature. I wanted to exercise this functionality anyway, since I do expect to use it in the future when I add disks to the array.

UPDATE (7/04): RAIDCore customer support (Jeff Huber) suggests that I could have accomplished this step more easily by right clicking on the failed disk, then selecting "initialize and set as ... spare".


Once I did a “commit” of this transformation, the online conversion from the critical disk-missing array to the new three-disk configuration commenced.


In about eight hours, the rebuild was finished, and all was well again with the array.

For a significant portion of the time the rebuild was taking place, I continued to use the system for text editing, email, and browsing. I also periodically played some video content off the array, watching and listening for glitches or hiccups. I didn’t observe any. The cpu utilization during the rebuild was generally about 20%.

I should also mention that during the rebuild, I had the array’s write caching turned off. I did this for a couple of reasons. I wanted to know a worst case scenario for a rebuild, and I wanted to see how it impacted what I would consider typical use conditions if I left write caching off all the time.

I’ve done some informal testing since the original configuration testing, and the write caching seems to get about a 10-fold increase in write speed. I haven’t taken the time to see if it provides a similar speedup in the rebuild, but I suspect it would have a substantial impact.

The Cost

At the outset, I wanted this to be an affordable solution. I didn’t expect it to be low cost. One giant disk would have been much less expensive.

RAIDCore RC4852 350.00
SuperMicro CSE-M35T 150.00
(3) Samsung SP1614C 345.00
(8) 1m SATA cables 48.00
(2) SilenX fans 35.00
5-bay cabinet 7.00
TOTAL 935.00

As built $/GB for 320 GB capacity 2.92

Expanding to the full 5-disk capacity of the pictured cabinet and chassis combination drops the price per GB considerably:

(2) additional Samsung disks 230.00
TOTAL 1165.00

$/GB for 640 GB capacity 1.82

So, this hasn’t been a cheap endeavor, relative to current “raw” disk prices. The Samsung disk nets at about $0.72 per Gigabyte, a rather astounding price. For the convenience of a single large filesystem and the resiliency the RAID5 offers, the $1.82/GB price seems pretty reasonable. In a full eight-disk configuration, the total increases to $1680, and the price per GB drops to about $1.50. By comparison, if I were building a similar capacity out of seven pairs of RAID0 disks, I’d have a bigger space problem, and the total cost would be above $2K. Most standalone or “bundled” RAID5 arrays are at least twice that price.

One of places I could have economized was the hot-swap chassis. As often as a disk might fail, or be added to the array, the hot-swap capability is a bit of a luxury. But like using a good tool, there can be convenience and comfort just when you need it.