I don't really play horror games (because I'm a massive wuss) but there are a few games that definitely could scare you.
Potzato: The butcher in Diablo was awe inspiring.
I totally second this. All the bosses in Diablo and Diablo 2 are pretty bad, mainly because if you die you have to try and run in and retrieve your corpse and equipment without dying. But the Butcher was particularly bad because he was the first boss you encountered (as far as I remember, it's been years since I played it). I think the only way me and my brother ever killed him was by locking him on the other side of some bars and shooting at him.
I remember Half Life being pretty scary. Low ammo and head crabs flinging themselves at you from dark corners, too small to hit easily so you ended up wasting all your ammo in a fit of panic. Then you had to kill everything with your crowbar.
Eternal Darkness on the Gamecube was also pretty scary. Not so much the monsters, just the spooky atmosphere and the slowly going insane. Very unnerving.
Not quite on the same level of fear, more tension I suppose, but back when I used to play Counterstrike when you got down to the last two or three players and you're slowly creeping around the map trying to find your enemy before they find you, that was pretty tense.