I had this problem. I fixed it by unchecking gpu scaling box in digital panel properties in catalyst.

I ran into this problem on a 2012 Macbook Pro with Nvidia GeForce GT 650M; none of the compatibility mode options, resolution changes, or scaling mode changes helped. I fixed the problem by checking the box next to 'override the scaling mode set by games and programs' in Nvidia Control Panel.