Build markets, with all the extras like a tent or a bench or signs and that stuff, that will help you get the prestige points to unlock the lords manor. And you have to deliver requested items to your people, not to the king or to the church, so you have enough points to unlock it, too :)