The most annoying thing about the game for me is how European powers, even when at peace with you, cluster their dragoons around your colonies. Right now England has a dozen dragoons just hanging out around my colonies, blocking my roads, and sitting on top of squares that I'd like to improve.
When I send a scout to their city to demand that they remove their forces from around my colonies, they ay that their forces protect valid interests, and will not be moved.
Any tips on getting them to move, short of declaring war? I really don't want to spend the horses and muskets it would take to kill them all.
Build a road out of the colony when you set it up, and then station one/two soldier units outside on the road, and set them to fortify (or sentry). That way, you are always guaranteed a way in and out of the colony if annoying forces do start showing up.
If you are not at war with the other power, just having them sat outside your colony wont affect whether you can upgrade the area or not - only if you are at war with them and they decide to fortify on that tile. Annoyingly, sometimes the AI will go to war with you unexpectedly and fortify anyway, so you still need good colony defenses.
One surefire way of stopping the problem in the first place is to have an aggressive policy against nearby European powers from as early on as you can possibly manage - preferably before they stockade their first settlement. I always try to get a ship (preferably a privateer) with 2 dragoons ready to attack the nearest power by around 1510 or so. This can be done most games, even on Viceroy level. If you can draw first blood, you put them on the back heel from the start, and they are much less likely to bother you as the middle game progresses. It also gives you a much needed early population boost if you can capture their first settlement or two, and as long as it isn't stockaded, you can always ship all of the captured colonists back to your main colonies, and abandon it. If the European power starts a new settlement, simply rinse and repeat until they slow their empire building right down, which they eventually must do.
I tend to use a privateer for this mission, as with any luck, you might be able to sink their ship as well, slowing their progress down even further - especially the English who only start with a caravel.
Whatever your policy towards the other Europeans early on, one good tip is to have a scout right next to a colony of the annoying power. that is pestering your colonies. If you do need to attack one of their forces to get them away from your colony, attack it, and then have your scout negotiate peace BEFORE the end of your turn (this works 100% of the time if you have Benjamin Franklin as a Founding Father). This way, the AI wont attack you during their turn, so you always have a free hit on their forces, without giving them the chance to hit back :)
It doesn't help your particular problem in this case, but hopefully it will give some food for thought when planning a future strategy.