Avogadro6: If you grant them a fief they'll become Lords and leave your party forever. That's a major downside right there...

Essentially, they become vassals, raise their own army, pursue their own goals unless you order them around, defect to other kingdoms if you don't keep them happy etc. Just like any regular Lord... except they'll be much weaker, so you'll probably want to train and equip them properly for the task beforehand (and that of course takes a lot of time, time you could have spent building solid relationships with other lords to try and make them defect later).
Another minor disadvantage is that giving land to a companion who is not a noble - and most of them aren't, check the wiki for more info- is a rather scandalous thing to do in Calradia, and it will make you lose a decent amount of rep with all the other Lords of every faction.

Honestly, outside of roleplaying purposes, I never found much of a point in the whole thing; maybe if you find yourself in a situation where you need more vassals ASAP to support you (e.g. you just started your own kingdom) then yes, it is an advantage, but then again you lose a companion permanently, and in my opinion you're much better off getting some regular Lords to defect to your faction.