Fairfox: @ least document it for others
I found it yesterday! There's a guard patrolling up and down the stairs in the Far Tower*. He stops at the very bottom of the tower, just in front of a door to the armory. He holds the key on his belt. I just took him down with a slime jar and grabbed the key. It also will open the armory door that connects with the courtyard.
* Well, I guess strictly speaking, the Far Tower if the semi-ruined one covered in vegetation. The map that covers this part is still the one called "Far Tower". I mean the tower that connects to the fortress that you access crossing the upper bridge.