[Guide] Hardware Transcoding: The JDM way! QuickSync and NVENC

Nvidia NVENC with GTX / Quadro Series GPUs

As stated in the introduction, adding a Nvidia card with NVENC to your server is easy. Most people know how to install a GPU. To get it working, all you have to do is install the latest Nvidia driver for your card, and enable hardware transcoding inside of Plex.

Before you dig through drawers to find an old GPU or start thinking about dividing your SLI in half to speed up your server, there are a ton of things to consider.

Performance and image quality between GTX900/Quadro M and GTX1000/Quadro P series are very similar with only marginal improvements. RTX/Turing have significant NVENC improvements over the previous generations, but come at a higher cost.

Requirements:

  • Full hardware transcoding (decoding + encoding) requires Windows 8 / Windows 10 (all editions)
  • Free PCI-E slot in your server (some cards take up double or triple width)
  • PCI-E 6 or 8 pin power connectors(s) (if your GPU requires it)
  • Maxwell / Pascal generation GPUs (or higher) are highly recommended, earlier generations have extremely limited support
    • GTX 900 / 1000 series
    • Quadro M / P series
    • Refer to this table (click “complete list” below the first chart)
  • A monitor or dummy plug must be plugged into the GPU at all times for hardware transcoding to work
  • Hardware transcoding requires some CPU power to process the data, even when decoding and encoding are taking place on the GPU. A slow CPU may artificially limit your GPU’s capabilities.

Restrictions:

  • All GTX series cards are limited to a maximum of 2 simultaneous transcodes
    • There are hacked drivers for Linux and hacked Dockers that unlock GTX cards, but that will not be covered here.
      • There is no unlock solution for Windows
  • Most Quadros have no hard limit on number of transcodes. In reality, the practical limit is around 17-22 simultaneous transcodes, depending on supporting hardware and software configuration.
  • NVENC support is different for each model of card, so make sure it supports the formats you will be using
  • Using more than one card is not currently supported
  • Decoding is not supported on any other OS besides Windows 8 / Windows 10 (all editions)
    • In case of other OS, only the encoding process will use your GPU. The decoding process will still use your CPU.
      • The hacked drivers do not affect the decoding/encoding support, only the maximum number of simultaneous sessions.

Pros:

  • Easy drop-in hardware installation
  • Easy software configuration
  • Use your existing server
  • Potentially very powerful with the correct hardware and software combination

Cons:

  • Often expensive
  • Can take up valuable space in a server
  • Certain servers (i.e. Supermicro rackmounts) do not have support for PCI-E 6/8-pin power
  • Software and hardware limitations
  • There are more than a few outstanding issues and bugs
  • Hardware transcoding still uses the CPU.
  • Hardware transcoding will not fix a slow server.

Known issues and bugs:

  • [common] Random BSOD on Windows (the only OS that supports decoding+encoding)
  • [common] Other applications using the GPU can interfere with transcoding processes causing them to delay, hang, or crash.
  • [common] Plex ultimately decides whether to use hardware transcoding or not. Sometimes it decides it doesn’t want to.
  • [uncommon] Starting a file via direct play or direct stream and switching to transcoding can crash the entire server
  • [uncommon] Switching bitrates while transcoding can cause the stream to endlessly buffer
  • [rare] Inconsistent but noticeable artifacts, especially during scenes with high motion

Other notes:

  • Plex will automatically fall back to CPU transcoding when it deems necessary, or when it’s out of transcoding slots on the GPU.
  • There’s basically no reason to buy or use anything higher than a GTX 1050. It supports all of the same formats as the more expensive cards, and since you’ll be limited to 2 transcodes anyway, there’s no reason to spend more on a “faster” card.
  • Plex will use QuickSync first in case of both a iGPU and Nvidia GPU installed.

Recommendations:

  • The Quadro P2000 is probably the ideal Quadro card. It supports all of the latest formats and has no hard limit on transcoding sessions. However, it’s still quite expensive anywhere from $275-$400.
2 Likes