Annoying fellow. Keep chopping away at him and can't make a damned dent in his health (keeps rejuving with every drum hit). I've pretty much resigned myself to running away from him until he stops following.
One amusing(?) anecdote: One of these chaps followed me into the Barracks. When I went back later, he had pretty much killed all the soldiers there save for two and the General. All I have to say is, "Poor Isolde".

Try the freeze spell along with a poison weapon. Load the poison weapon skill, right click, load the freeze skill (lvl 2 or better seems to work best), dispatch all his buddies, freeze him and hack away ! Still takes some hacking . . . =^ ) . . . Later, when you are at a much higher level, the drummers are easy.
Edit: lure his guard away as his drum restores their health if I remember correctly, once he is alone, freeze away.

One thing I noticed is the drummer seldom beats his drum if you used a range attack like a bow. Get your bow skills up and you'll probably drop them in a short time. If other troops are around, however, they'll probably drum away again which makes killing them or other nearby troops a nightmare. I soon learnt the merits of using a decent bow when I took on the orc hordes and the drummers.

In my latest game, I found the Orc Drummer and Orc's in general to be vulnerable to the Lightning skill. The skill works as a ranged weapon and, with a higher level . . . takes em out.

I've used poison to deal with them. Poison is very very effective against all orc's :).

Seconding this. That's how I always deal with the large Orcs in blue armor (their actual name escapes me right now...)

I use a bow pretty much exclusively, as I am putting all my first hundred plus points into agility and strength (even though I'm a mage) and don't have the health to stand up to any hand to hand. The drummer regenerates very frequently regardless. Enough that the drummer is a major ordeal.
I've found that there are some areas where you can get him turning back and forth to face you when you put an obstacle between you. That gives you more chance of getting his health down with a bow more than a third, at which point you can start chugging potions and hitting him with spells. Hitting him with spells before he gets his health down significantly is just a mana-bursting waste of time.
Probably much different the higher levels you get.

edit a week later:

I got a bow with a speed boost of 4, which makes a huge difference with these guys. I used to use a bow that hit much harder, but at the standard speed of 15. Doing much more damage looks to not be anywhere near as helpful as simply shooting more often. Now the drummers aren't a very big deal anymore.