I found a sort-of workaround for the problem if anyone else sees this and is having it.
The best you can do is set the resolution to 1024x768 and set other settings to your liking, then go to Documents\My Games\Myst V End Of Ages\init\graphics.ini and go to the file's properties. From there, check "Read Only" so that when you change your screen resolution later, this file doesn't get reset.
The downside here is that you will need to change your screen resolution every single time you want to play the game. The positive, however, is that all of the graphical quality settings now actually stick and you shouldn't have potato PC texture quality anymore.
It's really stupid that I have to do this, but it works. And yes, I also just confirmed that it is happening on the disc version on this PC, so I really don't even think it's a Cyan problem. It seems more like a Microsoft or an AMD problem.