[Official] Tdarr QuickSync-Enabled HEVC Encoding Plugin

Technology Software
1

I talked about the QuickSync Tdarr plugin I was working on in the Donator-chat channel on Discord this morning, and mentioned I was still working out a couple bugs. I played around with it a bit more and now the two files that failed previously completed successfully, so now I’m posting it in here if you want to test it on your own media collections.

This plugin does one thing, it re-encodes the video stream into h.265 HEVC. All other streams (audio, subtitle, attachment, data, and metadata) are copied to the output file without modifications. I’ve tested it with h.264, mpeg2, and vc1 videos, and in each case the resulting output looks to my eyes virtually the same as the input, at approximately half the file size, both on my monitor and my TV. I know that when you re-encode video you always lose some quality, which is why I added an optional filter to the plugin to only work on remuxes, as well as an optional minimum bitrate filter, so you can make sure that you don’t re-encode a shitty source.

Performance-wise, on my HP 290 I can transcode a single h.264 remux at roughly double speed. Adding a second concurrent transcode doesn’t really slow down the first, but when I go higher than that, individual transcodes start slowing down. The most I tested was 8 at the same time, and the average transcode speed was 0.5x. The good news is, you can run Tdarr transcodes and Plex transcodes simultaneously. I haven’t tested the combined limits yet, but I was able to run 2 Tdarr transcodes and three Plex transcodes with buffering issues in Plex.

This plugin uses ffmpeg, so you don’t need the QSV container, the haveagitgag/tdarr_aio container is all you need. Update: As of 5/24/2020 the containers have been merged together, so you should switch to the haveagitgat/tdarr container at your convenience. If you’ve never used Tdarr before, please read the documentation before you let this loose on your entire library, because Tdarr is designed to replace your files with the output. For testing purposes, you may want to use the Output Folder option in the libraries so it copies the output there instead of overwriting the original.

I’m just positing this on my Github gist for now. Please keep this link private for now, I want to do a bit more testing and polish the code a bit before I submit an official pull request on the Tdarr_Plugins project. To install, download the zip file from the gist link below and paste the single file inside the zip file (Tdarr_Plugin_Mthr_VaapiHEVCTranscode.js) in your Tdarr local plugins folder. If you have any questions, comments or concerns, please let me know either via reply here or in the Discord.

You can download the file here:

Update 05/28/2020 - My pull request was accepted, so you can now find this plugin in the community plugins. Look for plugin ID Tdarr_Plugin_Mthr_VaapiHEVCTranscode

And finally, as with most software, this plugin is being provided “AS IS”. I make no warranties, express or implied, and hereby disclaim all implied warranties, including any warranty of merchantability and warranty of fitness for a particular purpose.

4 Likes
2

Thanks for this I started setting this up yesterday and got to a point where I could trans code but found that I was going to need to go custom plug-in so that we could enable QSV . I really appreciate the share.

3

I made two very minor updates to this plugin. First, I removed the debugging messages to clean up the code before I submit a pull request to the official Tdarr repo. Second, I turned off the option to requeue the file after the plugin is finished, since it isn’t necessary to run the plugin a second time. I also moved this thread to the software category.

1 Like
4

I’m really interested in downloading only remuxes and using your plugin to transcode to h265. Haven’t yet played with your plugin yet but am curious if you have suggestions for automating the workflow such that new downloads via NZBget/sonarr/radarr get automatically transcoded and replaced. Ideally the original file could be played via Plex while the transcoding is in progress. Thoughts?

5

Assuming you’re using linux for your file storage and linux containers, you should be able to just point Tdarr at your library folder(s) and let it run. It will happily churn through your media converting it as necessary until all files have been processed. On the slim chance that Plex is using a file that’s being re-encoded, it’s not a problem because of how linux works with files. Plex will have access to the original file for as long as it needs it, and Tdarr will be able to use it simultaneously. Even after Tdarr has finished processing the file and moves the new one into place, Plex will still have a reference to the old file, which will still be accessible until Plex no longer needs it. The old file won’t be completely removed until no processes are using it.

6

Sounds perfect. I’m using Unraid. I assume what you said applies? I also assume that when the file gets transcoded, Plex’s video preview thumbnails will need to be regenerated (I currently have it set to generate thumbnails immediately, not just as a schedule task). Do you know if Plex will recognize the transcoded file as a new library item and thus generate the video thumbnails again? I suppose if i had to i could have it only run as a scheduled task and as long as the transcoding finishes before the scheduled tasks run that it will generate thumbnails using the new transcoded file.

7

I haven’t used the thumbnail features of Plex in production, but I would imagine that since the content itself isn’t actually changing the thumbnails shouldn’t really need to change either. I could probably stand up a test instance of Plex with a couple files to see what happens, but won’t be able to do anything like that until late tonight. One other thing you might want to consider is letting Tdarr run on your newly acquired media before it’s even added to the Plex library. SpaceInvader One did a 2 part video series on using HandBrake to transcode new files, using an intermediary folder for finished downloads as the HandBrake input folder and mapping the HandBrake output folder for Sonarr/Radarr to import. You could use Tdarr as a drop-in replacement for HandBrake and follow the rest of his steps basically verbatim. Here’s a link to the first video in the series: https://youtu.be/yibUWakxR18

8

Agreed that the thumbnails wouldn’t NEED to change I’m just guessing that Plex won’t know the difference and that once it sees the underlying video file has changed (different size etc) that it will either regenerate the thumbnails or no longer use the thumbnails it previously created.

Agreed that the simplest method would be to have Tdarr run before adding to my Plex library and that could work for things that download from lists but I really want the ability for me and my shared Plex users to add things via Ombi and have it download and be ready to watch within ~5 minutes. Having Tdarr run sounds like it would add an hour to that process.

Will check out the video!

9

Things were a little slower at work than I thought today, so I spun up a test Plex server to see how it would react to a file being replaced. I made a test library with two movies that I had previously re-encoded in h.265 HEVC. One movie I started with h.264, the other I started with the h.265 file.

After scanning and processing the thumbnails, I started playback of each movie and skipped around to verify the thumbnails showed up properly in the dashboard, which they did. Then, I stopped the movies and swapped them with the other versions. When I attempted to play both movies after swapping them, they wouldn’t play back at all. They just sat at the loading screen with the movie background and the orb spinning until I cancelled playback.

Next, I scanned the library and tried to play the movies again. The scanning process was almost instantaneous, and there was no re-creation of the thumbnails. This time, both movies played properly again, and the thumbnails still showed properly in the dashboard. While the second movie (the one that was now h.264) was playing, I removed the file and copied the h.265 version back into the folder. As I mentioned in my previous post, Plex continued to play the deleted file with no issues. I was able to pause, rewind, fast forward, and skip around with no issues in playback. As soon as I stopped playback and attempted to play the file again, it sat at the loading screen like it did the previous time I swapped files.

One final time, I rescanned files, and again the scan was done almost instantly, no thumbnail regeneration, and both videos played properly. So, if you want to put your files in the library immediately and re-encode them later, it is possible, but you will need to do a rescan of the library before you can play the updated titles. So, in your case, since I believe your files are stored locally, you can enable the options to scan my library automatically and run a partial scan when changes are detected under the Library settings. I tested this specific scenario on my test setup and it worked pretty well. About 3-5 seconds after I swapped one file, I saw that video (and only that video) refresh very briefly, and once refreshed the video played properly. So there will be a very brief (~5 second) window after the old file is removed and the new file is added where someone wouldn’t be able to play that specific file properly in Plex, but otherwise it should be pretty seamless.

For everyone else reading this reply, if you are using an NFS export to share your media with your Plex server, the partial scan when changes are detected option doesn’t work properly over a networked file systems, so a little third-party help will be needed. I haven’t used it personally, but I do see that Tdarr has a plugin to use plex_autoscan to notify Plex of the updated files. Or, you can follow the suggestion in my previous post of converting the media before you import it.

1 Like
10

First off, a thank you to Mthrboard for a solid video transcoding plugin.

I’ve been testing tdarr for a few weeks or so, and running it on my library for about a week. While running multiple transcode workers seems desirable on its face, I ran Glances to see the effects of each change in my configuration/setup.

My stack consists of 9 plugins, which include bringing video to front for better codec identification, standardizing subtitles, standardizing audio, transcoding video to HEVC and reordering streams. I found that running higher numbers of transcode workers was fast as long as not more than two workers was transcoding video at the same time. But because the video transcode is the longest step in my stack, running more than two transcodes invariably means that there will be times when every worker is transcoding video at once.

After testing five transcode workers, then four, then three, and then two, I decided to go with with just one. Surprisingly, at this point, I find running just one transcode worker is slightly faster than running two. But since I’ve been running one for only a few hours, I’m going to let it run longer to see how many GB’s of media are transcoded in 24 hours and then 48 hours.

Because I’m running 9 discrete plugins, while versatile and quick to setup, because I only needed to modify a couple plugins to my needs, not write my own from scratch, multiple plugins adds delay at each step where the plugin identifies and makes changes to the file, because the changes are written to disk and the modified file is re-queued. So it can take multiple re-queues of the same file to complete all the steps in the stack. This means that the files are written to the array after each change. So, there would be a sizable time savings by having a single plugin that handles all or at least most of the tasks, handling all the tasks in write cache before performing a final write to the array.

11

I’ve tried so many times, but I don’t understand this sentence.

12

I think he’s saying anytime the file changes other plug-ins scan it. So only outputing the file once would save him time and work better.

13

This might be easier on the eyes. Lol.

I’m running 9 discrete plugins. Plugins are versatile and quick to setup. I only needed to modify a couple Community plugins to my needs, not write my own from scratch. But multiple plugins add delay at each step where the plugin identifies and makes changes to the file, because after each change the file is written to the array and then re-queued for another pass.

1 Like
14

I agree with you completely, it would be much easier if Tdarr would cycle through all of the plugins in your stack before copying the final output to the destination folder. The good news is that’s on the developer’s to-do list. I don’t know exactly when he will get to it, there are a lot of other items on the to-do list as well, but it should be updated eventually.

In the meantime, if you’re comfortable with JavaScript, maybe take a look at your stack and see if some of the plugins could be combined. You said you have one that moves video to the front, as well as another that reorders the streams - those could probably be done at the same time. Audio and subtitle standardization could maybe be merged as well?

Also, make sure the plugins aren’t running when they don’t need to. A properly structured plugin should examine the relevant ffProbe data and set processFile to false if the file already matches the output of whatever the plugin was supposed to do.

15

I’m thinking about changing the hardware in my unRaid box and would like to make it so I can use it as an encoding box as well to convert files to 265.

What CPU would you recommend for this?

16

Hey Peeps, is this setup still the recommended way to go or is it worth considering unmaniac? I’m new to both.