I had two spare Character Points (waiting for the Velorien's blessing, when I'd have enough Dexterity to max Pick locks and... something, I don't remember what I was planning to do with the second one), so experimented with it myself after saving a game - three ranks didn't do the job, but four did (maybe it also depends on Charisma, and that's why three ranks worked for you?).
As for pickpocketing or killing him, I didn't have enough ranks in pickpocket to steal the gem (they were waiting for the blessing as well), and fighting him got all my good followers to throw a fit (also considering the attitudes of majority of mages inside, I'd sooner want to kill them, than him :P ).
Anyway, I got the blessing now, and so I'm now off to finish various remaining quests etc, instead of saving the world, the Big Bad will wait until I get there, anyway. Well, probably, but seeing as I'm on it for eight in game years (really, started as 15 years old half-orc, now I'm 23) and he's still sitting where he is (avoiding spoilers), I will probably be fine. :)
So I will get back to that quest, eventually. The guy can camp in front of Tulla for a couple more weeks or monhs, I guess.
I'm kind of tempted to spend enough in game years for a half-orc to die of old age and see if that happens, but no idea what would I be doing for so long a time (probably get bored out of my mind and quit or finish the game anyway :P ). And it probably wouldn't happen anyway.