That's part of the problem this particular encounter removes most your abilities ASIDE from the swordsman ones and even some of those. It even gets rid of your potion effects. If you haven't specced for extra vigor, improved blocking and riposte this fight might be nigh on impossible.
I was going to open a new thread, after ragequitting yesterday due to the Draug combat.
First of all, I loved TW1, I'm a huge fan (to the extent of being a fanboy), and I'm loving TW2 (except for a bunch of issues that, fortunately, don't drag too much the experience in the "shitty" experience).
BUT, said that, the Draug combat and whole "third" part of the Mist section has been thought by an imbecile, plain and simple.
I've specced FULL Alchemy, with just 3 talents in the Sword tree, I always relied all game (playing on HARD) on potions + adrenaline + Berserker, just to discover that, when you finally get to fight the Draug as Geralt, you don't have any pots active (even if you drank it before entering the Mist, as I did) AND you can't drink any in any part in that third section.
I didn't got problems fighting with the ghosts, I don't find what OP said as a problem (it was really easy to get rid of the enemies at the start of the section), but fighting the Draug with just Quen lvl 1, 3 Vigor and just Specter Oil on my silver sword and nothing else is near impossible, you have just to hope to avoid all the damage that the Draug can do, roll roll roll and hit him enough times to get him down, using a "lvl oneish" character since all MY talents are substantially down the drain. And it's boring that I have EVEN to start again fighting with the knight instead of starting directly against the Draug (autosave anyone?).
I think I'll need to switch at Easy for this, but I'm REALLY pissed off by this choice, it's really poor (just to don't say anything worse) gameplay design.