Anniversary-ish: Build Help and What else do I need?

Im reviving this thread but maybe I should have made a new one just for visibility…

In 2021 I plan to refurbish or replace the above build and id like some advice before I spend the cash. I would like to do it for the following reasons (in order of importance):

  1. It encodes videos with ffmpeg, handbrake, and transcodes in plex very slowly and maxes out resources (no Quicksync).
  2. It can handle about 7 plex streams max. The plex experience off it is its most important function but it doesnt do it very well.
  3. Its incredibly loud particularly when under encoding/transcoding load.
  4. I can only get the motherboard to accept 28 GB of DDR3 memory. I get a post error everytime I attempt to put more in. I have beat my head against that wall for hours and just gave up and I have ~100 GB of good DDR3 ECC memory sitting on the shelf. This lack of memory has kept me from loading this up with virtual machines as I originally planned.
  5. It draws a lot of power. The wife says it represents about $35 a month in our electricity bill.

I know I can fix #3 with the replacement of the PS units to the 920W SQ variants but I have been apprehensive to do so only because they are quite costly now on ebay and I would need to replace all 3. The money really isnt the concern as much as it would do nothing to fix the rest and I worry Id be throwing good money after bad.

I guess what I would really like to do is replace the motherboard and processors inside the case because that would allow me to keep running all 24 drives in the same unraid configuration I currently have. I currently have 24 cores but with the machine being memory starved I cant really use them to great effect other than encoding/transcoding so I could lose some cores and still be happy. I only have one VM on there I would want to ensure I retain.

I am looking at options that offer quicksync and enough cores. I am seeing deals on core i9s and core i7s with motherboards in the under $500 range but then I would need to buy new DD4 memory and still upgrade those power supplies. I am expecting that move would cost around $1k and I dont know what motherboard I could put in that case and what modifications I would require.

Alternatively I have looked at making an Otis machine and simply separating plex from the rest of the NAS and I suspect that is what you will recommend. I guess my apprehension on that solution is it doesnt address any of the other problems I have with the NAS.

I cant say that I like either of those plans so I am here to shop ideas. How should I spend my 2021 IT budget to accomplish these goals?