I'll admit I've been guilty a couple of times since I first came here.. I downvoted two or three times in the past when I didn't like what someone had said to me. An interesting feature of the downrep button is that it makes the post get hidden so you don't have to see it, so there's that psychological idea that downvoting a particular post will make it go away for you.
Normally, however, I only click the red - button when I think someone is truly being a jerk or probably needs to leave. As has been said, when someone is trolling or spamming and such.
Youtube has a problem with this, actually, as you'll see some otherwise good videos (or simply comments) downvoted because somebody didn't like what was shown or said for.. well, flimsy reasons. I uploaded a very short clip of my accomplishing something in a ROM once, and people downvoted it because it didn't look "nice enough" and the audio was out of sync. The point was to simply show it and say, "hey, I pulled it off", not to post up a freaking work of art.