Setting up media server using Ubuntu and SnapRaid

Resources Software
2

Install and Setup SnapRAID

https://zackreed.me/setting-up-SnapRAID-on-ubuntu/ - Official from SnapRAID but is cumbersome. Some folks have made it easier with a script and a docker image to install everything for you. So lets use: The Perfect Media Server 2017 | LinuxServer.io to do this.

Summary: We create a docker to get and build the SnapRAID, then install the package it made, the docker container kills itself when done.

apt install git
git clone https://github.com/lord-carlos/docker-SnapRAID.git
cd docker-SnapRAID/
chmod +x build.sh
sudo ./build.sh
cd build/
dpkg -i SnapRAID*.deb

Test that it worked by typing SnapRAID -v, you may have to use sudo SnapRAID -v.

To configure SnapRAID, we need to edit /etc/SnapRAID.conf, that is done by using nano again: sudo nano /etc/SnapRAID.conf should have some default stuff. Make it look like https://zackreed.me/setting-up-SnapRAID-on-ubuntu/ says.

# Example configuration for SnapRAID

# Defines the file to use as parity storage
# It must NOT be in a data disk
# Format: "parity FILE_PATH"
parity /mnt/parity1/SnapRAID.parity

# Defines the files to use as content list
# You can use multiple specification to store more copies
# You must have least one copy for each parity file plus one. Some more don't hurt
# They can be in the disks used for data, parity or boot,
# but each file must be in a different disk
# Format: "content FILE_PATH"
content /var/SnapRAID.content
content /mnt/disk1/.SnapRAID.content
content /mnt/disk2/.SnapRAID.content
content /mnt/disk3/.SnapRAID.content
content /mnt/disk4/.SnapRAID.content

# Defines the data disks to use
# The order is relevant for parity, do not change it
# Format: "disk DISK_NAME DISK_MOUNT_POINT"
disk d1 /mnt/disk1/
disk d2 /mnt/disk2/
disk d3 /mnt/disk3/
disk d4 /mnt/disk4/

# Excludes hidden files and directories (uncomment to enable).
#nohidden

# Defines files and directories to exclude
# Remember that all the paths are relative at the mount points
# Format: "exclude FILE"
# Format: "exclude DIR/"
# Format: "exclude /PATH/FILE"
# Format: "exclude /PATH/DIR/"
exclude *.bak
exclude *.unrecoverable
exclude /tmp/
exclude /lost+found/
exclude snapshots/
exclude *.!sync
exclude .AppleDouble
exclude ._AppleDouble
exclude .DS_Store
exclude ._.DS_Store
exclude .Thumbs.db
exclude .fseventsd
exclude .Spotlight-V100
exclude .TemporaryItems
exclude .Trashes
exclude .AppleDB
# Adding exclusion of downloads since we get a bunch of partial files here:
exclude /Downloads/
exclude *.backup\~
exclude *.partial\~
exclude node_modules/
exclude .git/

This docker installs left some trash, I followed https://stackoverflow.com/questions/46381888/docker-images-shows-image-docker-rmi-says-no-such-image-or-reference-doe to remove it.

Schedule SnapRAID

The Perfect Media Server 2017 | LinuxServer.io - Explains that there is a script to do this for us. Its old and uses python2 and a default python conf file to run.

  1. Create a folder /opt/snaptraid-runner/ (see link)
  2. Download sample conf and py script (see link) into folder
  3. Setup Conf:
    1. Threshold to 250
    2. Email, probably just error long term, put your email in
    3. Smtp - using the blog’s link has the proper gmail settings of port 465, ssl true. Use your gmail account, then you need to get an app password for the password. Or see Mailgun below.
    4. I left scrub settings default to true, 12 and 10

To schedule make sure you use the root crontab. Either sudo -i before or sudo crontab -e. Add the proper scheduling you want to do. Mine looks like:

00 08 * * * python2 /opt/snapraid-runner/snapraid-runner.py -c /opt/snapraid-runner/snapraid-runner.conf

My conf file looks like:

[snapraid]
; path to the snapraid executable
executable = /usr/local/bin/snapraid
; path to the snapraid config to be used
config = /etc/snapraid.conf
; abort operation if there are more deletes than this, set to -1 to disable
deletethreshold = 250

[logging]
; logfile to write to, leave empty to disable
file = snapraid.log
; maximum logfile size in KiB, leave empty for infinite
maxsize = 5000

[email]
; when to send an email, comma-separated list of [success, error]
sendon = success,error
; set to false to get full program output via email
short = true
subject = [SnapRAID] Status Report:
from = <CUSTOM MAIL ADDRESS>
to = <YOUR EMAIL ADDRESS>

[smtp]
host = smtp.mailgun.org
; leave empty for default port
port = 465
; set to "true" to activate
ssl = true
user = <CUSTOM MAIL ADDRESS>
password = <PASSWORD>

[scrub]
; set to true to run scrub after sync
enabled = true
percentage = 12
older-than = 10

Running SnapRAID manually
The SnapRAID documentation shows how to use a lot of the commands by hand, but I’ll go over the high level here.

  • sudo snapraid diff will tell you what has changed…
  • sudo snapraid sync will run a backup. The script does this daily, but it may fail if you’ve deleted a lot of content. It’s handy to be able to manually run when needed. Note - Sync can be slow, so you want your SSH connection to stay alive during the whole process
  • See documentation for recovery options.

Samba for FileSharing

I copied the old file to /etc/samba/smb.orig, then I added without changing anything:

[storage]
   comment = Storage on <SERVER NAME>
   path = /mnt/storage
   browseable = yes
   read only = no
   guest ok = yes
   hide unreadable = yes

Then restart the server using sudo systemctl restart smbd

You should be able to browse in windows under the network, but this won’t let you write or change files. https://www.howtogeek.com/howto/16196/how-to-disconnect-non-mapped-unc-path-drives-in-windows/

You may need to open up a command prompt on windows and use net use to delete the existing one and re-add it using the new credentials.

Connect at \\SERVERNAME\Sharename

I later followed the same steps for the opt folder, naming it configopt

Adding more drives later

To add a drive later, I have to follow the following steps:

  1. Physically install the drive
  2. Physically connect to the server (instead of SSH) to load the bios and ensure that it boots from the SSD instead of the newly installed drive.
  3. Format drive like in the formatting section above
  4. Add it to the FSTAB (and remount, a reboot also makes this easy)
  5. Add to SnapRAID.conf to include it into the array
  6. Run sudo snapraid sync to initialize the new drive into the array.

mergerFS Tools (Optional)

These are a suite of tools that help manipulate your mergerFS drives. You can remove duplicates, balance, or other strange moving tools. It’s not necessary, but sometimes can be useful.

Installation:

  1. cd ~ - to get to your home directory
  2. git clone https://github.com/trapexit/mergerfs-tools - Get the code on your local machine
  3. sudo apt install build-essential - Get the build tools you’ll need to install everything
  4. cd mergerfs-tools - Go into the code directory
  5. sudo make install - Build and install the commands

Move Files between drives
I had a strange situation. I had a couple nice 8tb drives, and a super old slow 2.5" drive I was adding. In addition to media, I have some old static backup files that I only access once every blue moon. I decided that I wanted the backup content on the slow drive, as to leave more room on the big drives for fast media etc. So how do I within the mergerFS array do this properly?

  1. sudo cp -dR --preserve=mode,ownership,xattr /mnt/disk1/Pictures /mnt/disk2/ - Copy from one drive to the other without any date, makes using the tools easier.
  2. mergerfs.dedup -v -d newest /mnt/storage/Pictures/ - to simulate what files would be deleted.
  3. mergerfs.dedup -v -d -e newest /mnt/storage/Pictures/ - to actually run.

Thanks to GitHub - trapexit/mergerfs: a featureful union filesystem

Alias Setup

Do you want to do stupid things over and over…No? Do this:

  1. nano ~/.bash_profile
  2. Add the text you want and save.
  3. Run source ~/.bash_profile

My file looks like:

alias sudo='sudo '
alias df='df -h -x tmpfs -x devtmpfs'

You can add other shortcuts here. I often also add alias grep= grep --color=auto to mine.

3 Likes