[Guide] HDDs + multiple cache pools in Unraid

Self-Hosted & Services Unraid
,
1

image
Image credit: ServeTheHome

Overview

Hey everyone, this will be a quick overview of how I use HDDs with multiple cache pools. A few notes before we begin:

  1. I’m not suggesting ditch your SSD cache entirely, I haven’t done that on any of my servers. I use both HDD and SSD in cache pools.
  2. SSD endurance is something that should be on everyone’s radar. If you can limit writing to your SSD, it will last longer. Sure, there are high endurance SSDs (and I always recommend them), but they are often more expensive than “normal” SSDs. Also, there’s no point in writing certain files to a SSD cache if the performance will be the exact same writing to a HDD cache instead.
    • HDDs generally sequentially read/write much faster than GbE, meaning this is ideal for large files such as media.
    • Smaller 10k/15k RPM HDDs in RAID0 are plenty fast for unpack directories. They are cheap and will prevent SSD wear.
    • Reducing the responsibilities of your SSD cache will increase its performance.
  3. HDDs are cheaper per TB than SSDs by a huge margin. Hell, the 1TB drives I use in the guide were free with the HP S01 I recently bought.
  4. This won’t work for every use-case, so adapt it to your personal needs.
  5. This is a good time to review how cache works on Unraid.
    Check out this video: https://www.youtube.com/watch?v=X0hX1cDP_1w
    Quiz yourself: do you know what the following mean?
    • Cache: Yes
    • Cache: No
    • Cache: Prefer
    • Cache: Only

Hardware recommendations:

Literally any HDD will work for bulk file ingest, even standard consumer ones. I use 1TB WD Blues for Server 1, because that’s what I had handy and that server is SATA only. I don’t store anything important there, period.

Another option is cheap, small, but fast 2.5" SAS 10k/15k RPM drives. These are great for use in single drive, BTRFS RAID0 (stripe) or RAID1 (mirror) depending on what you want to use them for. Many people have been using them in RAID0 for an unpack directory, as to not thrash their main SSD.

Server 1

I have a lot of hard drives in this server. I spin down drives when not in use to save power. Honestly the server isn’t used that much, it’s basically a media archive. Out of 28 HDD (+2 parity) only about 3-5 are spinning daily. I use multiple HDD cache pools + CA Mover Tuning plugin to reduce drive spin up even further. I keep the HDD cache pools spinning, and there’s one for each of my shares (another reason to not just use a single large share called “media”).

image
image3324×790 263 KB

This is how the share is configured, we’ll use TV as an example:

image
image2487×59 5.44 KB

Basically, anything that was ingested on the server within the past 30 days stays on the HDD cache pools. After the file has been there for 30 days, it gets moved to the main array and “archived”. It’s still accessible obviously, but it’s on the main array. Here’s the CA Mover Tuning config that I use:

image
image2099×1657 216 KB

As I said in the overview, I use this in conjunction with SSD cache. The following shares should be configured as Prefer:SSDcache, and should never live on the array:

  • appdata (apps will be extremely slow on the array)
  • domains (want a slow VM? put it on the array)
  • system (ever have your web interface go down during a parity check?)

Server 2

This server is my Unraid Gaming machine. I don’t do anything fancy with CA Mover Tuning, but I do have a HDD cache pool. This one is a pair of WD Red 6TB in BTRFS RAID 1 (mirror). I use it as large file ingestion, but I move content off of it daily. I also use it to store a large library of games that I don’t want on my array, by using Prefer:HDDcache on that share. It also has a VDisk for my gaming VM that is for large bulk game storage on steam - games that aren’t affected by loading time, basically.

image
image3620×472 91.6 KB

image
image2526×61 5.57 KB

Conclusion

HDDs are useful tools in cache pools due to their size:cost ratio. You won’t burn them up by writing large files to them constantly like you would an SSD. This is a relatively new feature of Unraid, and I see a lot of people aren’t using it yet. HDD cache isn’t ideal for every situation, but most would benefit from adding at least one to their setup.

If you have any questions, feel free to ask!

15 Likes
4

So, because i’m not an unraid user, I need you to dumb it down even further for me.

The main goals of using a beater 1TB drive as a cache is to:

  1. leave the big drives not spinning to save some electricity
  2. save wear and tear on the big drives?
5

Hi @easyrhino,

Happy to.

First, let’s go over how cache works in Unraid.

  • Cache can be SSD or HDD based.
  • You can have multiple drives in a single cache pool.
    Typically, multiple cache drives are configured in RAID 0 (speed) or RAID 1 (redundancy). Sometimes they are configured in multi-disk (JBOD), which simply adds them together with no speed or redundancy benefit.
  • You can have multiple cache pools (as long as you’re running Unraid version 6.9.X or higher)
  • Each share can only utilize one cache pool at a time. This is why I recommend splitting your file types into many shares, instead of using one large “media” share.
  • Each cache pool can service one or many shares.
  • Writing directly to the array in Unraid should be avoided when possible. When writing directly to the array, parity data is calculated and written in real time. This is a CPU and disk intensive process.
    Cache is used to mitigate the negative effects of real-time parity calculation.
  • Cache pools are for generally for write-cache only.
    Files are only read off of the cache pool if the files exist on the cache. Unraid cannot pull files onto the cache dynamically for read caching like storage-tiering file systems such as ZFS or Storage Spaces.
    • This guide covers how to use cache pools as a pseudo read cache. It’s has similar functionality and benefits, but it’s not totally the same.
  • A plugin called “Unassigned Devices” was popular for many years. This allowed users to use single drives with applications like VMs or just as simple storage directories.
    The functionality of this plugin can largely be replaced by using multiple cache pools. This is preferable because there are more options with cache pools, such as assigning direct interaction with shares and multiple drive configurations.

Shares can be set up to interact with cache in 4 ways.

  • Cache: No - Cache is not used.
    This is the default option for shares.
    When a file is written to the share, it is written directly to the array and parity is calculated in real time. (slow)
  • Cache: Yes - Cache is used
    Most of the time, this is the option that should be used.
    When a file is written to the share, it is written to the cache. On a schedule, the file is moved off of cache onto the array (with a process called the “mover”), and parity is calculated in real time… Generally this scheduled process takes place during off hours, for example: 4:00 AM every day.
    Parity is not calculated until the mover runs.
  • Cache: Prefer - Cache is used.
    This is a good option for shares that should live on cache, like System and Appdata.
    When a file is written to the share, it is written to the cache. If the cache fills up, the overflow will write to the array directly, and parity is calculated in real time. If the cache then frees up some space, the mover will move files from the array onto the cache, using the same schedule as “Cache: Yes”.
  • Cache: Only - Cache is used.
    There’s little reason to use this option. Cache: Prefer is always better.
    When a file is written to the share, it is written to the cache. If the cache fills up, the write operation will stop. Parity calculation never takes place.

TLDR:

  • Set media shares to Cache: Yes and use a HDD cache.
    • Use the mover tuning plugin if you’d like to set up a pseudo read cache.
  • Set System/Appdata shares to Cache: Prefer and use a SSD cache.
  • Set media unpack shares to Cache: Prefer and use a SSD cache (preferably a separate cache from System/Appdata), or RAID 0 HDD cache (my preference).

Regarding your questions:

  1. Yes, spindown isn’t important to everyone - but it’s important to some of us. Especially those with large amounts of drives.
  2. I suppose it does save wear and tear on the main drives, but only a little bit. It’s mostly to save wear on the cache SSD(s), as there’s no performance difference between writing a large file (ex: movie) to the cache HDD vs a SSD.
5 Likes
6

I’m moving into Unraid more and more each day, removing data off Synology, Xpenology, Windows, and TrueNAS systems.

I use the UA plugin to create a mnt for the mariadb container that can be read on my network (ie mounted by a thinclient, possibly my HA-Pi in the future). I’m not sure if that’s best-practice or not?

7

aahhh thanks for the unusually detailed explanation JDM. it was epic. I didn’t know that writing to the unraid drives was ‘expensive’ in terms of parity and speed. so that makes more sense.

incidentally i’m noodling with the idea of, on my openmediavault system, of using mergerfs to create a “write cache” SSD… it’s in the mergerfs man page and might be worth a shot.

1 Like
9

Hi there, sorry resurrecting an old post JJ :slight_smile:
Quick question I have 2 SMR drives I am retiring from my array as they seem to slow things down somewhat. I was thinking of keeping them in 2 seperate cache drives for various backups and things I wont be touching too often like photo zip backup folders, maybe old pc game install files, CA backups etc from my main unraid(as well as backed up elsewhere off-site).
Is there any obvios reason why I shouldnt do this?
Thank you in advance
Lunch

10

That’s exactly what I have running, but to be honest, I wouldn’t use a single non-parity or raid protected drive for backups. I’ve using mine as actual cache drives for my various shares as suggested in the OP.

11

I have been using UNRaid since 2009, actually still have that old box lol. I mainly use it for my media, and emby. After reading this article, my way of doing things is so outdated lol. I am grateful to JDM, for teaching this 'Ole Marine something new.

I am going to have to redo my setup now, but I want to make sure I am understanding you correctly. Instead of having Movies, TVShows under Media, they should be independent and should use a HDD cache, and Appdata, and System can be on a NVME cache - prefer. Did I get that right?

1 Like
12

I’ve read this post a few times and have been debating if HDDs as a cache is actually a good idea.

I understand that raw read speed won’t matter on gigabit internet but SSDs have crazy high IOPS which would actually help with write latency and iowait quite a bit more.

Has anyone actually tried using HDDs as a cache (even in raid0) and noticed how they perform?

13

I’ve been using a single 2TB spinner as a cache pool for my downloads and media shares for about a year now with no issues. I still have a separate SSD cache pool for Docker appdata, VM images and a couple other shares that benefit from SSD speed, but I see no need to waste an SSD for a meda cache pool or downloads that are only going to live on the drive for a day.

14

yes. read point 1 again

15

Interesting. I’m very split because I use an SSD for my downloads but I’m always running out.

It’s unintuitive to me that a HDD would perform that well but perhaps I should give it a shot.

16

You really think I don’t at least test, let alone use this myself?

17

Oh that’s not what I meant to imply but I was hoping to get some performance numbers or limits.

(I love your work, followed your NK1 guide to build my first NAS ever).

18

@ JDM_WAAAT
I put in an order at BitDeals based on your recommendation. It’s paid for, it’s been almost 3 weeks and nothing has shipped AND nobody is replying to my tickets. Here’s an idea, don’t recommend them anymore.

19

I haven’t been recommending them due to their shipping issues.

I fully encourage you to do a chargeback.