Windows 10 is not exactly known for it’s greatness as a Plex Server…I’ve never personally tried it so I can’t comment. My immediate concern would be the 6gb of ram under a decent load…Windows 10 by itself is probably using 4gb. 5 or 6 transcodes later…it’s probably inching closer to max capacity. If Windows 10 is sitting on a spinning hard drive…that’s another area of slowness. Obviously the gpu will take care of a lot of this problem too.
The 1660 is what $200-240 bucks? So you are looking at adding a cheap ssd (as cache) which can be had for like $20 bucks. Now you are at $240 to 260 thereabouts being put into a very long in the tooth computer. You could probably accomplish the same goals with a less expensive Nvidia gpu, horsepower here is not as important as just getting modern gear.
This really just comes down to how comfortable you are with diving into Linux or in general working on computers. Some people can pick it up and others are like hell no.
The little HP Slim that many others and I myself have purchased is $119 (or less)…you can add some ram for $40 bucks or whatever, an ssd is still $20 bucks. Then there’s the hdmi dummy plugs for the headless server and a few other things. Basically plan on spending about $175-$195.
Ubuntu is free and works way better for Plex. If you must…you could probably continue to use Windows 10 that it comes with…maybe move it over to an SSD. Otherwise you’ll probably need someone to help you network share to Linux from Windows. I don’t think an NFS share on Windows is going to fly. I would imagine it would involve enabling smb 1.0 on Windows and creating a share…typing it to the fstab through whatever means it takes to do that.