Moving Apartments - Server Build Do-Over Time! - Request for Build-Plan Critique

[Nick]

Hi everyone,

For those of you on the discord you’ve probably already heard me babbling about this, but shortly after completing my first NK4 build earlier this year, my wife and I are switching apartments…which means it’s time to re-evaluate everything!

Since I’m effectively breaking everything down to move it, this is the perfect opportunity to change aspects of the build, improve pain points, and buy new hardware!

So what worked, what didn’t, and what could I change for the better? Let’s find out!

[Nick]

1. What hardware am I running right now? What’s the Overall Architecture?

Note: This is mostly lifted from my NK4 build log (linked above). The only changes were the addition of a QSV Box (HP ProDesk 400 G4) and a CyberPower UPS

Enjoy my lovely drawing skills. Here’s my home office setup. The blocks on the left are this IKEA shelf if you need an idea of scale.

Currently, my NK4 (finally built so it makes the drawing ) is the black PC on the bottom.

Here’s a more recent photo. Pardon the mess!

Working backwards from the network, here’s what we’re working with and how it’s set up:

• My TP-Link Archer C7 v5 router is not pictured, but on the other side of the apartment in the living room. It’s an AC1750 router, so we get pretty solid speeds, but we’re limited by our 330M down/30M up internet. It’s cheap, but we’re bottlenecked on the upload side
• Working back from the network we have a TP-Link Range Extender underneath my desk, which provides an AC1750 Wi-Fi uplink to the cluster of nonsense on the left. The extender has a single 1GbE port
• That 1GbE port is connected to a set of 2 TP-Link TL-SG105 5-Port GbE Switches and 1 TP-Link TL-SG1005P 5-Port PoE GBe Switch. The switches are patched together with small runs of Ethernet cable, and I have 4 PoE + 6 regular GbE ports

So that’s the network. Now for the cluster. This project originally started as just a BeagleBone Black, but grew to eventually be a small SBC cluster. Here’s what we got:

• An NK4. This build is currently cased in a Cooler Master Elite 350, and is based around the X9SCM-F motherboard with a Xeon E3-1230v2. I have 10 HDDs + 6 SSDs in that case at the moment.
• An HP ProDesk 400 G4. This box is running as a QuickSync box running Plex. I added a 500GB NVMe SSD + a 500GB HDD
• A CyberPower UPS (900W)
• An ODROID-N2 w/ 4GB of RAM - Pictured underneath the Ethernet switches on the left with a single, blue Ethernet port and 4x Blue USB3.0 ports
• A BeagleBone Black Rev C - Pictured on the right side, middle of the stack of Pis with a single, blue Ethernet port)
• A Samsung SmartThings Hub (not pictured)
• A Philips Hue Bridge v2 (not-pictured)
• 3x Raspberry Pi 3B+ with the official PoE HAT (pictured on the right in the bottom of the stack of Pis with single, Orange PoE Ethernet port + 4x Grey USB 2.0 ports)
• 1x Raspberry Pi 4 w/ 4GB of RAM with the official PoE HAT (pictured on the right in the stack of Pis with a single, Orange PoE Ethernet port + 2x Blue USB 3.0 ports + 2x Grey USB 2.0 ports)
• 1x Raspberry Pi 4 w/ 4GB of RAM with a ClusterHAT v2.3 (pictured on the right, in the stack of Pis with a single, Blue Ethernet port + 2x Blue USB 3.0 ports + 2x Grey USB 2.0 ports)
• 4x Raspberry Pi Zero connected to the ClusterHAT and the top Pi 4 (pictured on top of the stack of Pis, arranged vertically)

That’s a long list. This project is capable of hosting everything I want and has been rock solid. The NK4 does the heavy lifting while the other nodes have been there mostly just for fun.

[Nick]

2. What Worked?

The Pi Cluster

So I’m going to start this section with a simple statement that everything I bought made sense at the time. All of this started with a $50 BeagleBone Black, and has been growing for the past 3 years. First it was a Pi, then a pair of Pis, then a cluster, then a NAS. Each time, what I was running was perfect for me. They were the perfect new project, new learning curve, and new area of the field to play around with. I wouldn’t change how I got to where I’m at now.

The biggest “what worked” for the Pi Cluster was that it let me inexpensively add cores/memory and add applications. I was able to easily scale up a set of services and build a low-power Docker Swarm cluster.

Additionally, the PoE HATs and ClusterHAT add-ons I bought, enabled a much more compact and interesting set-up than if I had just wired a couple Pis up to my router. I’m going to be keeping the PoE HATs, although I haven’t yet figured out what to do with the ClusterHAT. I might have outgrown this.

The NK4

Once I started to outgrow the Pi cluster, this build made complete sense. Everything “just works” now, and I don’t have to worry about downtime or a flaky SD card. The NK4 has handled everything I’ve thrown at it.

This project was fun! I never built a PC before, so this was huge and new and crazy to me. I feel like this build has a lot of longevity to it and will keep going for a while longer.

You’ll see later, but I have big plans for this build to keep it going and expand further.

About the Rest

There are plenty of other “what worked” items I could add, but really I wanted to highlight what I loved the most about this project so far. And in this case, that’s the learning/easy add-on capabilities that the Pis taught me + the solid reliability + power + fun that the NK4 gave me. I may add to the above, but I’d like to not write any more of a novel unless you all want me to.

[Nick]

3. What Didn’t Work?

Ok so this is fun. This build (if you couldn’t tell already) has been very, very ad-hoc. As such there’s a lot to poke holes in here. I’ll keep it to a couple key problem areas.

The project was so ad-hoc that each step required a complete re-invention of the underlying design

So, this is unfortunately a factor of how I got into this in the first place, but I started with a BeagleBone Black SBC. Great for learning. Unfortunately, very under powered. I needed more so I got more. Got a couple of Pis. Then a switch (5-port, what do I need more for?). Then some more Pis. Then an ODROID. Then another switch, but this time PoE capable. Then some PoE HATs for the Pis. Then a disk enclosure (USB to 4x3.5" SATA). Then I built an NK4. And bought a QSV box. And a CyberPower UPS.

If you can’t tell, this mess could have been addressed with proper planning and reasonable expectations. I never did any of that. The NK4 is still well within it’s bounds and I’m not maxed out, but since we’re moving now is a perfect time to re-scope everything. I could probably just relocate all of this and call it a day, but I’m out of disk bays - so sooner or later, I’m gonna need to upgrade the case on my NK4…and then what?

The Project is Not Near My Router/Modem

This wasn’t always the case. Originally all the Pis/ODROID/Disk Enclosures were next to the router. I used my Pis as a Pi-Hole DNS Blocker/Unbound Resolver. That all changed when the Fire Nation attacked when my wife wanted an electric fireplace/TV stand for Christmas. It looks beautiful, but prompted a full scale move into the bedroom which is where my work desk currently is. For the new apartment this is less of an issue, but one real problematic point has been network connectivity/cable management now that the router is not direct connected to everything.

The router had to stay by the modem which had to stay by the primary coax drop into our apartment. That is in the living room. The new place has multiple drops all over so during installation we’ll make sure to get this all in the same place. Nonetheless, it’s still worth mentioning.

A TON of stuff is only half-finished

This is a slight character defect on my part. I don’t like to finish projects. I get so excited with new stuff that I eventually get pulled away from finishing the thing I began in the first place. This list includes: MariaDB replication, actually starting my wordpress site, automating my “smart” apartment, auto power-down using NUT, etc.

This latest idea is a continuation of this defect in and of itself (ooh ooh shiny! Well we’re moving anyway right?!), but I hope this change now gives me enough flexibility to grow and avoid another “oh shoot I need a new case” moment in a year or two.

[Nick]

4. The New Plan

Here’s my new plan at the moment based on everything above:

Step 1: Co-locate everything.

A great deal of my problems with my current setup stem from the haphazard placement of everything. It’s why I have so many small switches. Why I have such a cluttered mess.

For this re-build, I’m going to move everything to be next to the incoming coax line in the office. This way I can have the modem, router, and every box that I want hardwired tied into them all in the same place. Hopefully this eliminates some bandwidth issues on the wifi as well.

Luckily, this is a free change - just moving stuff around.

Step 2. Time to Buy a Rack

If I wasn’t making a drastic move in the first place I would never have considered this, especially after just finishing my NK4 build, but I think it’s in my best interest to rackmount as much as I can. This has a few advantages/improvements over my current setup.

For one, it will declutter my office. It will also allow for the most efficient/varied expansion options (4U DAS, 1U OPNsense appliance, etc). All this WITHOUT consuming additional floor space once the initial investment is in place.

I’m thinking of either a 12U or 25U rack or cabinet. It depends on how much my wife will tolerate. I have a feeling I may get locked into buying a 12U or 25U cabinet for that reason, which dramatically raises the price.

Step 3. Re-case/Upgrade my existing hardware.

If I’m definitely going rackmount, I think the best thing to do is to re-case my existing hardware to maximize space. So to that end, the first thing up is the NK4.

For the NK4, this is moderately simple. I’m planning to buy a Rosewill RSV-L4000 (not L4500) 4U, 8-bay chassis. I know the usual recommendation is the L4500 (15-bay), but at the moment that is currently out of stock. Plus, I think that the L4000 will be perfect for this application since I actually could use the 5.25" slots. The plan here is to take my NK4, move all the parts over into the L4000. I have 3x5.25" adapters that each add 2x2.5" + 1x3.5" slots which will fit nicely in this case. Long-term I can either replace the 5.25s with a 4-bay 3.5" adapter, or up to 18 2.5" SSD slots.

The HP ProDesk 400 G4 and the CyberPower UPS at the moment are just going to sit on a shelf in the rack. I know I can get a 1U/2U UPS, but this will work fine for now, and while I could build an OTiS, I’m going to stick with this for the moment to maximize the money spent.

For the rest, this will be fun. The Pis are going to be pulled from their cases and re-racked in a 2U enclosure that I found here which looks awesome: Raspberry Pi Server Mark III – Uptime Lab

Which then leaves the networking equipment. I can repurpose one of the 1G switches and the WiFi repeater and connect them to the media equipment in the living room (Xbox, TV, Apple TV, etc), but I think since I’m going rackmount, it’s time to buy a 1U switch (maybe an Aruba S2500). I’m open to recommendations here, but for now the idea would be to go with something with 24-48 ports, with PoE to enable the Pi 2U rack.

I also want to build a 1U OPNSense box, but that’s a project for the future. When all of the above is set and done, I’ll have 10U used (including a shelf for the QSV Box + UPS.

New Parts	Cost	Notes
StarTech 12U/25U 4-Post Open Frame Rack	$210-$260 (Open Frame)	Wife could easily veto this
StarTech 12U/25U Enclosed Cabinet	$450+	Less Likely to get vetoed, way more expensive
Rosewill RSV-L4000 (8-Bay, 7 Fans) 4U Box	$120	Plan to use 3x5.25" initially to hold previously owned IcyDock 5.25 to 2x2.5 +1x3.5 adapters (effective capacity of 11x3.5" + 6x2.5")
Arctic P12 PWM PST 5-Pack	$31-$45	For the 4U Case. Depends on stock availability
Arctic P8 PWM PST 5-Pack	$26	For the 4U Case
Pi Server Mark III 2U Enclosure	$0*	*Brother has a 3D printer that can print this for me for the cost of materials. PoE HATS/Pis already owned
Aruba S2500 1U Switch	$120ish	Haven’t looked into this yet, would need PoE + ability to aggregate links so I can get my NK4 operating at 2Gbps
OPNSense 1U Box	200+*	Not planned yet
Rackmount 1U/2U UPS	200+*	Not planned yet. Current UPS works - vanity replacement/future expansion
Rackmount 1U/2U OTiS	200+*	Not planned yet. My HP ProDesk 400 G4 works, this would just be for fun if anything.

All told, I’m looking at a $387 to rackmount the NK4, and about $700 in future purchases/products offset by whatever I can flip my old non-rackmounted hardware for.

What do you think? Is this too much, inefficient or wasteful? Am I missing something?