I’m wanting to create a fileserver for my son who often works on film projects which bloat to 3-4tb sometimes.
The actual editing takes place on an onboard SSD but he needs a backup server which has decent amount of space and won’t take half a day to transfer 3tb of files.
I want to use softraid in the form of OMV. (Snapraid and mergerfs probably.) It needs to be hardware agnostic in case of breakdowns. Happy to source used kit.
Can this be done for around £500? I was thinking along the lines of an HP N50L or similar (Is there anything similar…?) but could they be upgraded to get better transfer speeds?
His workstation is a Windows machine. He doesn’t want to be running anything more off that and OMV is Debian in any case. He also wants it to be relatively portable if possible and not too noisy. Low power use…
2TB SSDs seem to go for around £130+. Three of those would be £390… It’s not enough space ultimately but it would be a start and the software would allow disks to be added I believe…?
How would we get that data on and off in reasonable time though…? It would need to be DAS I guess… Could a direct 20GBe be used…?
Your storage can be cheap/large/fast. Pick two. If you’re looking at backups since he’s operating on a local SSD, does it really need to be fast? Even if you were pushing an LSI 9207-8i (though you’d need the 8e), it will take close to 14 minutes to transfer 4TB. (Sata/SAS controllers tested (real world max throughput during parity check) - Storage Devices and Controllers - Unraid) So, yeah, DAS is your best contender, and remember to price out the card you need to put in the video editing machine.
The hardware to do the DAS without the hard drives could probably be found cheap enough. Maybe 30-40% of your budget. The hard drives are what’s going to tank your budget, especially if you want to back up more than one project.
Here’s a different thread of thought: Does the entire 4TB project change at one time? If not, only back up what’s changed and that will save you a massive amount of time.
your son may have a variety of desires that are conflicting, so he may need to choose.
how much total space do you think you need to have for backups? And what kind of networking infrastructure exists (or are you willing to install?) wifi only? ethernet? how fast?
For portability, maybe consider and external drive. like thunderbolt or the fastest USB available. Plug it in and do the backups. That’s the most portable. You could then work out an external storage to migrate it to a larger server later (I think OMV has an external storage plugin? haven’t used it).
If your networking is wifi or gigabit ethernet only (and borderline 2.5gb ethernet) you can just backup directly to hard drives, they can keep up. If you are talking 5gb or faster then you will need SSDs to absorb the writes if you want to go as fast as the network allows. a decent SATA SSD can keep up with 5gb, 10gb will require nvme or SAS (or raid 0) to keep up.
mergerfs can be configured as a “write cache”, and when with cron scripts to push data to the spinners. I eventually got it set up successfully when I had a 10gb connection between my workstation and server. (now it’s just wifi so it doesn’t matter).
Hello everyone. According to what I have read in forums, the Noctua models are the quietest fan. I want to install one on the Cwwk N100 seat but I don’t know the size to buy the right one (92 mm, 120 mm). You will not know what the appropriate fan size is or if you know a specific Noctua model.
Thanks for responding. Sorry I’ve been so slow…
I’m seeing this box as an interim backup, before projects get long-term archived to other disks or Backblaze/whatever.
We also don’t have to have full capacity right away, as long as the box has room to grow.
I’ve spent the last few weeks looking at old HP and Dell servers such as the HP Z420. If I have things right, this has 2 PCIe gen 3 slots and at least 3 drive bays and can be had used for £100. Then I’m seeing a PCI card able to hold 4 NVME drives, even if we only have budget for two at the moment, and a 6TB HDD. More HDDs to be added later.
Using something like MergerFs, we could set this up as two separate disks, the NVMEs and the HDDs. Projects could be backed up to the NVMEs very quickly a couple of times a day and then to the HDDs overnight. Eventually to the cloud or whatever once they are done.
This setup could expand as required couldn’t it? It would allow for quick retrieval of a project if needs be and a second backup for extra safety.
Thanks for the link to the DAS stuff. I’ll definitely have a look.
I think we’re along the same lines. See my reply to Omeganot above. I’m seeing this as a hybrid SSD/HDD box probably connected over USB4 for speed. Normal network/WiFi would definitely be far too slow.
I don’t think the budget will cover a fully fledged solution but it can get him started at least. With flexible software such as snapraid and mergerfs additional disks could be added easily.
I can’t see it having redundant power supplies etc, unfortunately but do you think it might work…?
Going all flash for an archive is, more or less, a waste of money. Assembling the storage device, either all flash or spinning drives, as some kind of interim solution, is also kind of a waste.
Is the intention to have a backup or archive? and does it actually have to be portable?
At least it sounds like he intends to do the active work on a local computer - what laptop even is it?
Could use some clarity or decisiveness on just what kind of solution is really being sought.
I’ll echo the comment on USB4 and starting with the work machine by @easyrhino. Is your son’s machine a desktop /workstation or a laptop? Do you already have USB4? Have you found a storage device that supports USB4 and can utilize it’s bandwidth? Do you have a plan for going PC to PC with USB4, because USB as a standard doesn’t typically like to do that. It wants to go PC to peripheral. I’m also guessing that HP Z420 doesn’t have USB4.
For PC to PC, your likely quickest solution is SAS. SAS2 is 6Gbps per lane, which is likely quicker than any network-based solution you would have. But also, you’d need SAS support on both devices, which likely means an expansion card.
You could probably max out 5 lanes at a time. NVMe drives are PCIe 4x devices, and the card they are on will max out at 8x (since there is at least one other card in play - the SAS HBA). As mergerfs isn’t a true RAID solution, it’s a union proxy over your drives, it probably doesn’t write to all drives simultaneously, nor does your copy program, so you’re likely writing a single file at a time. Snapraid also isn’t real software-based RAID, it’s parity bits added when a snapshot is performed, so it’s not giving you anything extra for speed. You’re essentially using mergerfs to get more storage space, which you’ll need if you’re going the expensive route and buying NVMe drives, but using one at a time. If you go spinners with hardware based RAID or software with something like ZFS, they could be configured in such a way as to write to them simultaneously. Not at the same speeds as NVMe, but not slow either.
Speaking of spinners, if you go with that HP Z420, you’re right in that you can put 3 in by default, and if you swap out the 3 5.25" bays you could add another 4 hard drives.
You can do any of these things. You just need to think of what you want to do and what you should do, and if those are the same thing.
The main editing is done on a desktop but the box I want to build is something like an interim backup, before final backup to the cloud let’s say.
It needs to be able to transfer a big project, let’s say 2TB, relatively quickly to and from the editing machine at the end of each day.
The first copy could then be backed up to the HDDs nightly and then finally to the cloud as the box fills up.
Does that make sense…?
The editing machine is a desktop with USB4.
I didn’t know that USB4 doesn’t like to link PC to PC but what defines a “Peripheral” though? A typical NAS drive is essentially a PC isn’t it…? Why isn’t the file server I’m wanting to put together a “Peripheral”?
No, the Z420 would need a USB card. Might be a gen3 to begin with…
I’m just suggesting mergerfs so that we only have one disk to deal with. Several smaller ones kind of defeats the object. Snapraid would probably only apply to the HDDs. I realise there’s no speed gain but it seems flexible in terms of adding storage to an existing setup.