I have finally completed my NAS Killer 4 and 8 Bay DAS builds. The final piece was tying the two boxes together with better cabling.
I didn’t know I’d wind up where I did when I started this journey last black Friday. I did set out to build a 12 Bay 48TB NAS but decided to go all in so I could back up the backup and create a real homelab experience so I added the 8 Bay 80TB DAS for a total of 128TB Raw. Part of the homelab experience was to learn a lot more about the windows server environment, play with storage spaces, play with hyper v’s and back up my home/boat audio/media/data servers. The source servers are all linux/zfs based devices. To solve the problem I need the hardware, plus a synchronization solution.
I started building the main 48TB array based on 2 adaptec raid cards at raid 6, 2 legs of 24TB. I wound up here because I didn’t want to lose the space to mirroring and I decided my initial storage spaces parity exercise was a failure based on performance so I got a card that could do both raid and hba but wound up implementing Raid 6. My performance on these arrays is consistently above 100m/s.
When I decided to add the DAS I figured I’d go in for the 10TB drives to give me some room to play with. Because I like pain, I wanted to try windows storage spaces again and bought an LSI 9201 card to use as an HBA adapter. I could never get more than 100mb/s even with caching supplied by a product called primocache. Parity just sucks. Mirroring was great and I got 300mb/s but gave up half my storage. I figured I’d buy another adaptec card and play some more with caching (increasing to 5 250gb drives). I needed more sata connections anyway and figured the card would act as a backup to the other 2. I just could not get the performance with windows storage spaces in parity mode, gave up and added a Two Ports Mini SAS SFF-8088 To SAS 36Pin SFF-8087 adapter which I hung off the adpatec card and moved the DAS to the adaptec with all caching. I used the card to implement another 8 drive Raid 6 array.
For caching I wound up using primocache and I’d say I like it a lot. Enough to add to my other array shortly. Since I had 128g of ram to begin with and only plan on running about 3 vm’s, I gave 24g of ram and the 1.25TB of disk to primocache. On the large array I am seeing 300-600mb/s transfer speeds up from around 100mb/s, sustained well over 300mb/s. On smaller files I was only hitting about 1mb/s, maybe a little more. That’s now sustained over 1mb/s, easily hitting 10bm/s.
My NAS/DAS now has 30TB+55TB useable. I am using 18TB in the source. It took 5 days to back this up at the beginning of this journey. In my final configuration, it appears to backup in 18hours so I am very pleased.
A couple of notes about the NAS. I used a rosewill 12 Bay hotswapple case with a supermicro x9 mb. I saved some good $ on the mb by buying used (processor/mb/ram together). I got a steal on the case which is no longer available. If I were to buy today I would probably go supermicro 846. The rosewill fans and cases were noisy and hot. I replaced the back fans with noctua and went with supermicro cpu fans (had a problem with space for the narrow noctura which was preferred) which solved all problems. Sticking 8 ssd’s in this case was no walk in the park.
A couple of notes about the DAS. I was worried about power on this and am happy to say its hung in there. I had trouble with heat initially to the point it shut down. I did replace the rear fan on initial build with a noctua but added 3 smaller noctua’s on a 3d printed card for additional cooling and split the power. This solved the problem completely. When I added the Two Ports Mini SAS SFF-8088 To SAS 36Pin SFF-8087 adapter to the adaptec card in the NAS I bought a second one and added it to the DAS and modded the bracket and added the plate to the DAS and swapped out the wiring to make this ultra secure.
The expense is an interesting question. So taking drives out of the equation and putting all raid/hba cards into the NAS cost, the NAS build wound up at $1295. Similarly with the DAS I wound up at $146. Drives are a ballbuster for cost. That’s why I went with 8 4TB’s to begin with. I now have an extra LSI 9201-16e which I will repurpose for future expansion to another DAS enclosure of some kind.
The NAS PARTS LIST:
SUPERMICRO X9DRi-F Motherboard 2x E5-2620 CPU 8GB RAM Heatsink I/O E-ATX IPMI $190
2 cpu coolers Supermicro CPU Heatsink Cooling for LGA2011 SNK-P0048AP4 $70
128g (16 X 8GB) PC3-10600R DDR3 1333 ECC Reg RAM Memory Supermicro X9DRi-LN4F+ $200
Rosewill RSV-L4312 - 4U Rackmount Server Case or Chassis, 12 x SATA / SAS Hot-swap Drives, 5 x Cooling Fans Included $110
2 Noctua NF-A8 PWM chromax.Black.swap, Premium Quiet Fan, 4-Pin (80mm, Black) $40
12 WD Red 4TB NAS Hard Disk Drive - 5400 RPM Class SATA 6Gb/s 64MB Cache 3.5 Inch - WD40EFRX $980
3 Adaptec 2274400-R SAS/SATA 6GBPS 1GB PCIe RAID Controller with Battery, (ASR-71605) $210
Two Ports Mini SAS SFF-8088 To SAS 36Pin SFF-8087 PCBA Female Adapter With PCI $25
Mini SAS 36P SFF-8643 to SAS 36P SFF-8087 Raid Data Server Cable 50cm US $40
CORSAIR RMx Series RM1000X 1000W 80 PLUS GOLD Haswell Ready Full Modular ATX12V & EPS12V Power Supply SLI and Crossfire Ready $200
2 CableCreation Internal HD Mini SAS Cable, Internal HD Mini SAS (SFF-8643 Host) - 4X SATA (Target) $30
2 Samsung 860 QVO 1TB Solid State Drive (MZ-76Q1T0) V-NAND, SATA 6Gb/s, Quality and Value Optimized SSD $176
1 2 tb Samsung SSD 860 EVO 2TB 2.5 Inch SATA III Internal SSD (MZ-76E2T0B/AM) $280
2 2 of Standard Full Height Bracket for Low Profile Cards $10
CNCT 19-Pin USB3.0 to USB2.0 Adapter Header Cable - Suitable for use with USB 2.0 $5
Intel Ethernet Server Adapter I340-T4 1Gbps RJ-45 Copper, PCI Express 2.0 x 4 Lane, OEM packaging $50
2 Rosewill RDRD-11003 2.5" SSD / HDD Mounting Kit for 3.5" Drive Bay with 60mm Fan $20
USB 3.0 20 Pin Male to Female USB 2.0 9 Pin Motherboard Male/Female Housing Adapter Cable $5
Grand Total $2641 w/out Drives $1205
The DAS PARTS LIST:
8 Bay Trayless Hard Drive 300 Watt Enclosure With 2 SCSI Cut Outs $90
8 WD Red 10TB NAS 5400RPM SATA 6 Gb/s, 256 MB 3.5" Hard Drive WD100EFAX $2000
5 Samsung 860 EVO 250GB 2.5 Inch SATA III Internal V-NAND SSD (MZ-76E250B/AM) $250
LSI 9201-16e 6G 16-lane external SAS HBA P20 IT Mode $50
2 CableDeconn Mini SAS 26P SFF 8088 Male to 4 SATA 4Pin Female 2m 6.6FT Cable $30
1 ARCTIC P12 Fan $8
SATA power cable: Monoprice 108794 24-Inch 4-Pin Molex Male to 4 15-Pin SATA II Female Power Cable Net Jacket (3 Pack) $13
Honeycomb 240mm Radiator Triple 80mm Fan Grill Computer Case Cooling Custom Mod $18
ARCTIC COOLING P8 Value Pack ACFAN00153A 80mm Pressure-optimised Case Fan (5-Pack) $20
CRJ 4-Pin Molex to 3 x 3-Pin PC Case Fan Sleeved Adapter Cable $7
Grand Total $2486 w/out Drives $236
