It's kind of a known problem -- especially since the AI cheats, so they know exactly when and where you built the new building without having to True Sight all over, and get through that mirror of protection. It's especially bad when you build mage and cleric temples: they go after those with a vengeance.

Try spreading them out -- one building per kingdom -- and if you desperately need one in a given area, try building more in another (ideally in a cluster) as "bait."