Hey, I’m running a Kubernetes cluster on Raspberry Pi 4s, but I keep running into issues because the SD card storage is so awfully slow. I could start building out faster storage for the Pis, or I could start looking at something else. I’m also considering the Turing Pi 2 when it comes out, but there are things I’d like to do that I can’t do on the Pis.
Current cluster:
4X Raspberry Pi 4, 4GB. PoE hats on each to reduce cable clutter and let me hard-restart a node remotely. 16GB microsd for OS, one 256GB USB SSD for the Kubernetes database. iSCSI on a dedicated device for persistent storage.
On this cluster, I’m running:
- bitwarden_rs
- nextcloud
- pihole
- rssbridge
- jellyfin
- homeassistant
- several attendant services, including cert-manager, metallb, postgres, fleet
I’m pretty happy with this setup, except:
- all my iscsi volumes are created by hand. I create the logical volume, add it to my iscsi tgtd config, generate chap auth secrets, and store those in the tgt config and the kubernetes database. i’ve scripted the process, so it’s about 5 minutes any time I want to add a volume. Something like rook-ceph or longhorn would be nicer, but I’d need faster disks.
- I’m running single-master. I’d need more SSDs to get to multi-master.
- I’m running Jellyfin, but it doesn’t re-encode anything except h.264 because that’s all the Pis can do, and even then it’s pretty rickety and slow. Lots of rebuffering, failed streams.
Some other things I’d like to do but can’t with the current setup:
- stream games with Steam Link. total non-starter on Pis. I use NVidia Geforce Now, but there’s some older games like Batman: Arkham City I’d like to play. I could see getting a mid-range GPU setup for older games, and keeping GFN for newer stuff.
- store more data locally. I’m paying about $35/mo to Backblaze for data storage. I could cut that down quite a bit if I stored more locally. ~6-8TB usable would probably keep me happy, I’d like 2-3 replicas per block. I’ll keep backing up to Backblaze, but only stuff I really care about, not using it as default storage for everything under the sun like I am now.
So, I’m looking at storage options for the Pis. In theory, the Turing Pi cluster should give me NVMe storage options on the pis, which would let me go multi-master and maybe share some storage with my Kubernetes pods. It doesn’t let me play games. The Turing Pi 2 isn’t priced yet, but the old version was $189 + 40 per compute module. The 2 is supposedly going to cost less, so let’s keep it the same as a worst-case: 189 + 4 x 40 = 349 + 100 for the NVMe = 449 + 3 10TB USB drives at 150 each = 899. I still can’t play games on that, and transcoding is still hitchy. I also have to find space on my UPS for 3 USB drives’ wall warts, which is a pain.
If I were looking through the new and/or used market, and I wanted to avoid a single point of failure, let’s say I got 2 x86 machines, 32GB RAM each, at least one of them capable of streaming Batman: Arkham City to another device on my LAN, .5TB NVMe for OS and fast storage, some spinning disks for bulk storage. Can I come up with something cheaper than the Turing setup? Where should I be looking? I’ve poked around unixsurplus.com, maybe a Dell R220? Do I need to chuck a real GPU into one of the units?
I’m imagining I’d run Proxmox or oVirt on each x86 machine, have a gaming VM, several Kubernetes VMs, and some extra capacity to screw around with.
I’m a bit space and power constrained, too. I’ve got a single 15 amp circut for my rack, and my rack is currently 8U high total, and 17 inches deep. I don’t have to stick with this, but it was free (built from scrap wood) and I’d want to figure the cost of a new rack into a new build.