qwixter: It's time to quit when you are no longer having fun. Being bad or good really shouldn't be that big of factor in single player.
I quit a game if I've
1. Spent an absurdly long amount of time on one level without beating it
2. I've tried every innovative or even obvious strategy I can think of, with no success, and I've tried my very best
3. I've tried taking a break and coming back to it several different times on different days
If all these things are true and I still am no closer to advancing, at the very least in the main story, then I basically say it's not worth my time and admit defeat. But I also second qwixter. If it's not any fun why bother playing any more?
EDIT: Sometimes I've quit a solid game because it was fun, but the first try on one game mode,(Original Starcraft: Skirmish mode) where I've had so much fun with similar games, was such a pain that I lost interest. The way I've heard it described is that "Starcraft makes you micromanage" to succeed. I agree, and I don't like doing that so SAYONORA.