Some last-minute questions before I finally complete my build

Builds [LGA1151,LGA1200] NAS Killer 6.0, Plex QSV Builds
1

I’m almost done building my first DIY NAS, and have a few questions before I boot it up for the first time.

Parts I have:

CPU: Intel i3 9100

Motherboard: Super Micro X11SCA-F

RAM: Kingston Server Premier 32 GB (2 x 16 GB) DDR4-3200 CL22 ECC Memory

Storage:

  • 1x Western Digital Red Pro 6 TB 3.5" 7200 RPM Internal Hard Drive
  • 2x Western Digital Red Plus 6 TB 3.5" 5400 RPM Internal Hard Drive
    Reason I have 1 Pro and 2 Plus is because I happened to get the Pro on sale for the same price as each Plus
    PSU: Corsair SF750 750 W 80+ Platinum Certified Fully Modular SFX Power Supply
  • 1x Transcend SSD370S 32 GB 2.5 Inch SATA (boot drive if I go with TrueNas)

Obviously, this doesn’t include stuff like fans or the case.

I plan on setting the 3 6TB HDDs in RAIDZ1.

My use-case is a Home NAS and eventually some Lab stuff as I start to learn more. To begin with, really just getting to know how it all works, setting up Plex and storing mostly music, and a few movies. I like to learn new things, so I will likely want to create various VMs to try things out (e.g. app and website development).

My question is around OS. I have gone back and forth between UnRaid, TrueNas and Proxmox. Currently, I’ve spent the longest time planning to go with TrueNas. However, while my plan is to use Scale, I’m not 100% sure I understand the difference between that and core.

What I’m looking for are people’s thoughts on which OS they would go with if they were in my shoes, and importantly, why.

Regarding TrueNas, if someone could concisely explain the pros and cons of Core and Scale respectively, that would be great too.

Lastly, I’ve never built a PC before and so don’t actually know what to do when it comes to booting it up for the first time. Do I need to go into the BIOS first? If so, what do I need to do there? Are there any tests I should be running before/after installing my OS of choice? How?

2

I’m a TrueNAS Scale user. I love it. Don’t use it.

I picked TrueNAS Scale over Core because of the better support for Docker, Kubernetes, and other apps, running on Debian over FreeBSD. I’m sitting on a combined 99TB of usable storage across three pools. Not too many more needs than what you’re describing. I also believe iXsystems is putting a lot more effort into Scale and not much at all into Core, so Scale is the more future-supported option. There’s no real reason to go Core unless you really want it.

The reason I don’t recommend TrueNAS for you is that you said this is the first PC you’ve built and you’re unsure about booting it up the first time. TrueNAS really isn’t that hard to install. It was remarkably easy, in fact. Configuring it isn’t really that hard either, but you should know what you’re doing. If something doesn’t work, how confident are you in digging in and figuring it out? I’ve built 4 servers and 7 PCs and I ran into steep learning curves with both the Supermicro board (new to me) and TrueNAS.

I would imagine Proxmox would be a bit of the same, but maybe harder for the configuration of storage, though you have perhaps a little more control there. TrueNAS and Unraid make it pretty simple through their UI, and while I don’t know if there’s an equivalent for Proxmox, it’s not designed for storage, it’s designed for virtualization (so, better if that’s your primary use case and storage is secondary). I have no Unraid experience, but you’ll find a lot of supporters of it here. It seems simpler than TrueNAS to get running.

But, what you really should do is try them out. Build the PC. Get it to boot, set up the BIOS. Learn how to use the IPMI. Then pick one of the free OSes (TrueNAS or Proxmox) and install it. Configure it. Play with it. Don’t load anything critical on it. Then delete it, and install the other. Learn the differences. Unraid has a monetary cost to it, so be sure you’d want that before sinking the money into it. If you can’t get TrueNAS or Proxmox to work or to your liking, maybe Unraid is better for you.

1 Like
3

A first-timer, plus with your stated work load, should probably go with Unraid for ease of use.

4

I definitely agree that TrueNAS (let alone running TrueNAS on Proxmox) would be quite a steep learning curve. That said, considering I will definitely be experimenting with various VMs further down the line as I build up my knowledge, I’m aware that a lot of users say that managing VMs with Proxmox is the superior way to go, compared to just using TrueNas. I’m not aware of how Unraid stacks up against TrueNas on Proxmox for features like that. Currently I’m leaning towards TrueNas on Proxmox, and accept that it will be a slow and difficult road to getting everything set up. I’m not planning on storing any important data on the system until I have everything set up in a way I feel confident about.

5

We do not recommend virtualizing your storage. It’s an unnecessary layer of abstraction and can complicate things if issues arise.
Unraid can run VMs just fine. It may not have any advanced management or special virtualization features for managing dozens of VMs but if you want to experiment with spinning up a few, it’s perfectly fine.

6

A few threads I’ve read through that convinced me TrueNas on Proxmox (or simply Proxmox) was the way:

https://www.reddit.com/r/homelab/s/tCFzpJiseb

https://www.reddit.com/r/Proxmox/s/NE47CHHgz3

https://www.reddit.com/r/HomeServer/s/w7cjnUikgv

https://www.reddit.com/r/unRAID/s/0CcxOR7Qoa

Importantly, I am very keen to use ZFS. I feel this will be a nice complement to the ECC memory I have. And the comments several people have made about LXC having the potential to affect other parts of the system outside of that container (compared to a VM being more truly “isolated”) is concerning, especially if I’m new to this whole game and will be prone to making mistakes.

7

I highly recommend that you use Unraid instead of the other choices.

8

Appreciate your input.

Perhaps you can reassure me on the points touched on in those reddit threads linked above, namely:

  1. A lot of people online claim that the ZFS implementation on Unraid is inferior to TrueNAS. Can you debunk this?

  2. People also seem to say that running containers and/or VMs in Unraid is not as feature-rich as in TrueNAS. If this is true, what use-cases would necessitate me moving up to TrueNAS for that kind of stuff, and what types of activities would Unraid allow for in fine fashion?

  3. Given that I’m going to be using ZFS (which is quite RAM-heavy, and I only have 32GB because of current budget constraints), if I combined this with Unraid (which is less performant than TrueNAS) what can I do to mitigate this double-whammy performance drop compared to going with TrueNAS/Proxmox with TrueNAS?

  4. If I went with Unraid, how best to utilise L2ARC in ZFS??

9
  1. Unraid uses its own storage scheme with a write cache, a main storage array of drives, and parity protection of that array. It doesn’t implement ZFS as part of its default scheme but does support adding drives with ZFS as sort of side or secondary storage. It’s supported, but not as front and center as other OSes.
  2. Unraid is fine, great even for Docker. There’s a huge number of containers in a super easy App store, plus you can add on whichever you might want if it isn’t offered.
  3. ZFS isn’t as RAM heavy as some think - TrueNAS uses a default of half your RAM as a write cache which is great but if you mean the 1 GB per 1 TB of storage, that’s not really true. Unraid’s storage scheme uses a separate drive or more than one, SSD or even HDD depending on usages, as write cache so a huge amount of RAM isn’t strictly required. This also potentially negates the need of things like L2ARC.
4 Likes
10

Thanks for that, that’s super clear.

I understand that Unraid implements its own storage scheme by default, but would it be possible for me to use ZFS on my main (and only) array? What about doing this would be less “complete” a use of ZFS compated to TrueNAS?

It seems like a lot of the TrueNAS loyalists go on about how ZFS on Unraid isn’t comprehensive, but all the comments regarding that date to before ZFS gained a more full implementation on Unraid. At the same time, however, I haven’t heard any Unraid devotees say with certainty that its possible to get the same experience as on TrueNAS (as far as ZFS goes). Overall a bit confused as to what I might be missing out on.

11

Yes various amounts of ZFS were possible on Unraid for a while which probably led to a bunch of people complaining it was incomplete - because it more or less was.

Array drives can be formatted for ZFS and leverage some of the capabilities of ZFS but basically, I think you’re going to be just fine using Unraid as it is. ZFS is cool but it’s a complication too, in Unraid or even as part of TrueNAS. Unraid is an excellent entry point to having a very capable, reliable home server.

12

What types of files are you storing? If it’s bulk media (video, photos, and audio) then Unraid will work excellently for that.

13

Yeah totally. It is indeed bulk media (to begin with) - mostly audio files, I have a pretty large music library (I think anyway).

I want to be able to:

  1. Stream my music (and small movie) library - both while at home on my laptop but also remotely on my mobile while I’m out.
  2. Store my family photo library and experiment with different open-source methods of managing family libraries.
  3. Some basic sharing capabilities for family members to view the photos.
  4. Storage and easy navigation of important documents, ebooks etc - also to access at home and remotely.
  5. Experimentation with things more typical of a “server” e.g. hosting websites and other things.
    5a. I am new to coding and development, but I see this becoming at least a small part of my tinkering time going forward. I want to be able to try things out and make mistakes without risking everything else going on on my NAS.

Currently I have 3x 6TB WD Red Plus drives, which I planned on configuring as RAIDZ1. Maybe I’ll hold off setting it up until I can afford a 4th 6TB drive in order to do RAIDZ2.

I also plan on using Backblaze for backups. Maybe further down the line a 12TB drive to act as a backup mirror for the main array, when budget permits.

14

Hosting things and making your data accessible to others safely is a separate chunk of learning to be done.
We don’t really recommend hosting sites in particular at all.

15

When you say hosting things is a separate chunk of learning do you mean:

  1. It’s a separate conversion to the one we’ve been having (ZFS), but ultimately is very much possible with Unraid.

or

  1. It’s a separate issue, but even then, this is not something you’d be able to do with Unraid, and would be possible with TrueNAS.
16

It’s entirely separate from discussing whether you use ZFS or not. It’s very much possible on Unraid. It’s not terribly doable on TrueNAS Core but might be possible and is probably possible on TrueNAS Scale but we still don’t recommend it and I have no idea how.

17

Hi and welcome to the server builds community.

There are a lot of different topics going on in this thread some of which could easily be their own stand alone discussions.

TrueNAS Core vs Scale

The main difference between the two boils down the the OS used. Core is built on FreeBSD and Scale is built on Linux specifically Debian Linux.

FreeBSD is very lightweight so you get great performance even on old hardware and ZFS support is very mature and rock solid stable. However the installed base and community is much smaller than Linux and as a result there are far less options for compatible software, new hardware takes much longer to get driver support, and there are a lot fewer resources out there for learning and support.

With Linux you get much better compatibility in terms of both software and hardware and there is a huge community and tons of resources out there to support it. ZFS support is a little less mature but it has come a long way.

I would probably only recommend Core if you only wanted to run a pure NAS without other services and you were using very old hardware. For the use case and hardware you mentioned, if you are certain you want to use TrueNAS, you would want to use Scale. However that brings us to the next topic.

Unraid

For the use case you mentioned I would highly recommend Unraid. Unraid is designed from the ground up to be a very user friendly NAS / Media Server. It is easy to get set up, provides a very user friendly interface for running docker containers and VMs, and the JBOD array setup is easier to use and expand than a ZFS RAID as well as being much safer in terms of not destroying your data by way of user error.

ZFS is an amazing file system. It is loaded with great enterprise level features and you can spin up 20 disks and completely saturate a 10 Gbps or faster network connection 24/7. However in reality that is never going to happen in a home server environment. A single disk will easily saturate a 1 Gbps link and you probably won’t even do that most of the time. The downside of ZFS is that with all those great features comes a very steep learning curve and the one thing ZFS does make very easy is to permanently eradicate all of your data with a single miss-typed command.

The only situation I would recommend TrueNAS over Unraid (for home use) would be if your main goal is to learn to use ZFS, however at the same time I would never recommend learning to use ZFS with data you care about or want to keep. Do your learning in some sort of sandbox environment with expendable data.

You can still use ZFS with Unraid although not quite the same way you would with TrueNAS. It works quite well for cache pools, and you can even use it to good effect in your main array, although it will be individual disks rather than zpools.

The Youtube user SpaceInvaderOne makes lots of great explainer videos on Unraid including the use of ZFS. If you do go with Unraid you should check them out: https://www.youtube.com/watch?v=rEAfX75nReg

First Steps after building your Server

You might need to go into the BIOS to reset settings to default or select a boot disk, but really the first thing you will want to do is get into the IPMI. There you can update your BIOS and BMC firmware and remotely control the system.

With a used motherboard, if it has not been reset to the defaults by the seller, that can be a bit tricky. The default IPMI password for your SuperMicro motherboard is ADMIN/ADMIN. However if there is a non default password you will need to boot into an OS (Probably Linux or FreeDOS) to run SuperMicro’s IPMI Tools to reset the password. Here is a good guide on that: Reset Supermicro IPMI Password to Default - Physical Access - STH

Hosting services publicly accessible on the internet

Pretty much any OS you choose will easily facilitate spinning VMs to host servers that you can use for app and web development. I highly recommend doing this on your local network. There is technically nothing stopping you from making these servers available to the internet at large, but as others have pointed out that is a whole can of worms that I would strongly recommend against opening.

There are many concerns with connecting servers on your home network to the internet. A typical residential internet connection is very low bandwidth, especially in terms of upload which is what matters the most for a webserver. You would be very vulnerable to ddos attacks which would effectively kill your internet connection to your home and might incur overage costs or the possibility of disconnection by your ISP. If you are not absolutely on top of security at all times you will also be susceptible having your server hacked by bots on the internet and any attacker would then have access to your entire home network.

A home server is great for doing development, but when you are ready to host services on the internet you would be much better off renting VMs from a hosting provider in an actual data center which will have a much faster and more secure network and things like built-in back up services and close to 100% up time.

Hope that helped answer some of your questions and whatever route you decide to take enjoy your new server!

1 Like
18

Few more things to consider:

  • Everything mentioned about the pros and cons of ZFS, especially the learning curve, is true. I created a ZFS pool with a single vdev, thought I could expand the pool with more disks in the same vdev so I didn’t have to adjust parity or lose another disk to a new vdev. Not true. That will change in the future, but still not a supported ZFS feature. I created a whole new pool, copied my data over (learned a lot about replication along the way), destroyed my pool (with all my important data) and created a new pool and vdev the way I thought it was going to work. It was a scary learning curve.
  • L2ARC - Maybe useful if you don’t have a lot of RAM, or if you have tons of data with a heavy usage load. Doesn’t seem to be your case. It’d probably be worth it to buy another 32GB of RAM over buying an additional drive (or multiple drives) for the L2ARC. I don’t see L2ARC drives in your spec list, so it seems you haven’t thought about where you actually put that cache.
  • Numerous people over here have recommended Unraid, whether with additional supporting information related to your questions or without. It seems you’re really hesitant to take that advice given the additional questions in your follow-up posts. Learning is great, and I’ll continue to encourage it. However, why don’t you learn by trying our your other options? Install TrueNAS. It’s free. Install Proxmox. It’s free. Install TrueNAS on Proxmox. Still free. You’re just investing time, and you say you plan to do that anyway to learn. Learn by doing. Then see if you wanna stick with any of these options or go to the Unraid recommendation.
  • TrueNAS on Proxmox - As many recommendations as there are to do it, there are just as many not to. The best case scenario is you’re passing raw drives directly through to the VM running TrueNAS, and even then there are a ton of things that can go wrong. But even if it was easy, the question would be, why? Proxmox supports ZFS directly. You can basically build your own pool of vdevs out manually in Proxmox and get the same thing you could from TrueNAS. TrueNAS is an appliance, in that you are installing a specifically configured set of services that make it easier to do all that setup and management that you could do manually in nearly any Linux environment.
1 Like