The signs dont work (I tried to no end) nor do swords (my original idea).

Grapeshot is what you want OP

Yes I know no where in the quest discription does it even remotely hit at this but thats how you destroy them

Grapeshot - Aye.

If you read the book on nekkers it and checked your knowledge or monster descriptions or quest journal - it quite clearly states that the best course of action is to collapse their tunnels with grapeshot bombs.