[Guide] HDDs + multiple cache pools in Unraid

image
Image credit: ServeTheHome

Overview

Hey everyone, this will be a quick overview of how I use HDDs with multiple cache pools. A few notes before we begin:

  1. I’m not suggesting ditch your SSD cache entirely, I haven’t done that on any of my servers. I use both HDD and SSD in cache pools.
  2. SSD endurance is something that should be on everyone’s radar. If you can limit writing to your SSD, it will last longer. Sure, there are high endurance SSDs (and I always recommend them), but they are often more expensive than “normal” SSDs. Also, there’s no point in writing certain files to a SSD cache if the performance will be the exact same writing to a HDD cache instead.
    • HDDs generally sequentially read/write much faster than GbE, meaning this is ideal for large files such as media.
    • Smaller 10k/15k RPM HDDs in RAID0 are plenty fast for unpack directories. They are cheap and will prevent SSD wear.
    • Reducing the responsibilities of your SSD cache will increase its performance.
  3. HDDs are cheaper per TB than SSDs by a huge margin. Hell, the 1TB drives I use in the guide were free with the HP S01 I recently bought.
  4. This won’t work for every use-case, so adapt it to your personal needs.
  5. This is a good time to review how cache works on Unraid.
    Check out this video: Unraid storage & operation basics - parity, cache, & your data - YouTube
    Quiz yourself: do you know what the following mean?
    • Cache: Yes
    • Cache: No
    • Cache: Prefer
    • Cache: Only

Hardware recommendations:

Literally any HDD will work for bulk file ingest, even standard consumer ones. I use 1TB WD Blues for Server 1, because that’s what I had handy and that server is SATA only. I don’t store anything important there, period.

Another option is cheap, small, but fast 2.5" SAS 10k/15k RPM drives. I encouraged BitDeals to put up a few 2-packs of 300GB to 1.8TB drives. These are great for use in single drive, BTRFS RAID0 (stripe) or RAID1 (mirror) depending on what you want to use them for. Many people have been using them in RAID0 for an unpack directory, as to not thrash their main SSD.

Server 1

I have a lot of hard drives in this server. I spin down drives when not in use to save power. Honestly the server isn’t used that much, it’s basically a media archive. Out of 28 HDD (+2 parity) only about 3-5 are spinning daily. I use multiple HDD cache pools + CA Mover Tuning plugin to reduce drive spin up even further. I keep the HDD cache pools spinning, and there’s one for each of my shares (another reason to not just use a single large share called “media”).

This is how the share is configured, we’ll use TV as an example:

Basically, anything that was ingested on the server within the past 30 days stays on the HDD cache pools. After the file has been there for 30 days, it gets moved to the main array and “archived”. It’s still accessible obviously, but it’s on the main array. Here’s the CA Mover Tuning config that I use:

As I said in the overview, I use this in conjunction with SSD cache. The following shares should be configured as Prefer:SSDcache, and should never live on the array:

  • appdata (apps will be extremely slow on the array)
  • domains (want a slow VM? put it on the array)
  • system (ever have your web interface go down during a parity check?)

Server 2

This server is my Unraid Gaming machine. I don’t do anything fancy with CA Mover Tuning, but I do have a HDD cache pool. This one is a pair of WD Red 6TB in BTRFS RAID 1 (mirror). I use it as large file ingestion, but I move content off of it daily. I also use it to store a large library of games that I don’t want on my array, by using Prefer:HDDcache on that share. It also has a VDisk for my gaming VM that is for large bulk game storage on steam - games that aren’t affected by loading time, basically.

Conclusion

HDDs are useful tools in cache pools due to their size:cost ratio. You won’t burn them up by writing large files to them constantly like you would an SSD. This is a relatively new feature of Unraid, and I see a lot of people aren’t using it yet. HDD cache isn’t ideal for every situation, but most would benefit from adding at least one to their setup.

If you have any questions, feel free to ask!

10 Likes

So, because i’m not an unraid user, I need you to dumb it down even further for me.

The main goals of using a beater 1TB drive as a cache is to:

  1. leave the big drives not spinning to save some electricity
  2. save wear and tear on the big drives?

Hi @easyrhino,

Happy to.

First, let’s go over how cache works in Unraid.

  • Cache can be SSD or HDD based.
  • You can have multiple drives in a single cache pool.
    Typically, multiple cache drives are configured in RAID 0 (speed) or RAID 1 (redundancy). Sometimes they are configured in multi-disk (JBOD), which simply adds them together with no speed or redundancy benefit.
  • You can have multiple cache pools (as long as you’re running Unraid version 6.9.X or higher)
  • Each share can only utilize one cache pool at a time. This is why I recommend splitting your file types into many shares, instead of using one large “media” share.
  • Each cache pool can service one or many shares.
  • Writing directly to the array in Unraid should be avoided when possible. When writing directly to the array, parity data is calculated and written in real time. This is a CPU and disk intensive process.
    Cache is used to mitigate the negative effects of real-time parity calculation.
  • Cache pools are for generally for write-cache only.
    Files are only read off of the cache pool if the files exist on the cache. Unraid cannot pull files onto the cache dynamically for read caching like storage-tiering file systems such as ZFS or Storage Spaces.
    • This guide covers how to use cache pools as a pseudo read cache. It’s has similar functionality and benefits, but it’s not totally the same.
  • A plugin called “Unassigned Devices” was popular for many years. This allowed users to use single drives with applications like VMs or just as simple storage directories.
    The functionality of this plugin can largely be replaced by using multiple cache pools. This is preferable because there are more options with cache pools, such as assigning direct interaction with shares and multiple drive configurations.

Shares can be set up to interact with cache in 4 ways.

  • Cache: No - Cache is not used.
    This is the default option for shares.
    When a file is written to the share, it is written directly to the array and parity is calculated in real time. (slow)
  • Cache: Yes - Cache is used
    Most of the time, this is the option that should be used.
    When a file is written to the share, it is written to the cache. On a schedule, the file is moved off of cache onto the array (with a process called the “mover”), and parity is calculated in real time… Generally this scheduled process takes place during off hours, for example: 4:00 AM every day.
    Parity is not calculated until the mover runs.
  • Cache: Prefer - Cache is used.
    This is a good option for shares that should live on cache, like System and Appdata.
    When a file is written to the share, it is written to the cache. If the cache fills up, the overflow will write to the array directly, and parity is calculated in real time. If the cache then frees up some space, the mover will move files from the array onto the cache, using the same schedule as “Cache: Yes”.
  • Cache: Only - Cache is used.
    There’s little reason to use this option. Cache: Prefer is always better.
    When a file is written to the share, it is written to the cache. If the cache fills up, the write operation will stop. Parity calculation never takes place.

TLDR:

  • Set media shares to Cache: Yes and use a HDD cache.
    • Use the mover tuning plugin if you’d like to set up a pseudo read cache.
  • Set System/Appdata shares to Cache: Prefer and use a SSD cache.
  • Set media unpack shares to Cache: Prefer and use a SSD cache (preferably a separate cache from System/Appdata), or RAID 0 HDD cache (my preference).

Regarding your questions:

  1. Yes, spindown isn’t important to everyone - but it’s important to some of us. Especially those with large amounts of drives.
  2. I suppose it does save wear and tear on the main drives, but only a little bit. It’s mostly to save wear on the cache SSD(s), as there’s no performance difference between writing a large file (ex: movie) to the cache HDD vs a SSD.

I’m moving into Unraid more and more each day, removing data off Synology, Xpenology, Windows, and TrueNAS systems.

I use the UA plugin to create a mnt for the mariadb container that can be read on my network (ie mounted by a thinclient, possibly my HA-Pi in the future). I’m not sure if that’s best-practice or not?

aahhh thanks for the unusually detailed explanation JDM. it was epic. I didn’t know that writing to the unraid drives was ‘expensive’ in terms of parity and speed. so that makes more sense.

incidentally i’m noodling with the idea of, on my openmediavault system, of using mergerfs to create a “write cache” SSD… it’s in the mergerfs man page and might be worth a shot.

1 Like