My mother dislikes him because he turned his nose up at a Pittsburgh dish on his show, acting like it was something horrible and wrong or some such. Dunno details, never saw that episode.
(Fried corn meal mush... think polenta or grits, but you let it cool and gel up, and then slice and fry it. Serve as breakfast with syrup.)
Personally I don't mind him. Don't really like him, but I don't hate him either. I just like shows that have people making different dishes around the country (or the world actually). He hasn't been annoying enough that I stopped watching his show (well, more like he isn't the reason I haven't been watching it).
Truth be told... I do kind of want to smack him when I see him wearing his sunglasses on the back of his head though.
Edit: To reply to the post made while i was typing:
Green_Hilltop: To answer the OP: People who hate him are just bullies or people who like hating on others, or who just jumped on the "everyone/it's popular to hate on X" bandwagon. Don't mind them, in most cases they're just ugly people (on the inside) you don't want to be around with or listen to. There's no reason to hate on anyone unless they have qualities that are seen universally as bad/evil or engage in such actions (bad/evil), whether on a small or large scale.
Not all who hate him fit into this. As was explained to me by some people that don't like him, he's very representative of Food Networks move from cooking shows to reality tv shows with cooking as a theme. Shows like his grocery games annoy a lot of people, me included. It has as much to do with cooking and food as my left nipple has to do with space exploration. And this person didn't seem to hate him so much as hate what he represented on Food Network.
And some include his Diner/drivins show with the reality tv shows that they dislike. (I don't, at least its about food, and not people acting like idiots.)