I had this problem before, but its been so long ago I can't remember how I fixed it. Here are a couple of suggestions:
1. Before you enter the castle, cast Invisibility spell. I think they will be friendly this way and you can get to the door.
2. You might have to go jail/pay your fines. I think fines can be paid at the City Counsel Building.
3. Might just have to wait a while. I mean in game time. Save your game and do things that pass a long amount of time like train/level up, do side quests, or just rest a lot.
Give these a try, I'm not 100 percent sure they will work.
Tried 1, I totally forgot about fines, I'll go try that. Thanks.