Welcome to Serverbuilds!
I recommend you check out the NK 6.0 Build Guide. It is pretty much exactly what you are describing; A value oriented NAS build that is based around 8th / 9th Gen (Coffee Lake) Intel processors that have a fairly modern iGPU for QSV transcoding.
The Coffee Lake (8th / 9th Gen) CPUs do support transcoding for H265; otherwise known as HEVC. Wikipedia has very useful table in the QSV Article that shows what formats are supported. Just scroll down a little to see the table. The relevant column for the NK 6.0 will be the (Kaby / Coffee / Comet / Whiskey) Lake column.
You will want a server with space for all your 3.5" drives. Besides your 3 data drives you will also want to consider an additional drive to allow parity protection. Having a parity drive means that if one of your drives fails the data on the drive will not be lost and you can replace the drive and restore the data to the new drive using the parity data. Depending on how full your existing 3 20TB drives are you will either want to get a 4th 20TB drive to use as a parity drive, or if you can move all your data onto 2 drives with space to spare then you could do that and use the 3rd as your parity drive.
Good luck with your build, let us know if you have any questions or run into issues along the way.