Playing with less pawns means you level up faster, gaining strength faster. If you play solo or with your main pawn instead of a party of 4, you'll gain levels faster, thus personally your character (and your main) will be stronger than playing with a party of 4.
It's up to you. It's easier with a party of 4 since there are more distractions for monsters to attack, letting you survive a bit easier. Of course pawns aren't too bright , but even just being there lets them be meat shields. Most players will play with a full party their first time, and when they are smarter and handle foes easier they usually will play solo or drop the 2 extra tag alongs and just keep their main pawn as a companion during the game.
If you play solo, when you do get your main pawn back out, it will level up very fast to catch up to your level, so if you play to level 100 and your main pawn is left back and is only level 50, don't worry he will be zipping up in experience and levels fast to catch up to match your level 100.
Up to you how you play, I played the normal game the first time with a full party, and then 2nd new game+ just played without the extra pawns on hard mode, so i'd get faster leveling and get personally stronger instead of doing it slower.
Thanks! Good advice, and good to know. My first time was also with a full party, so might go for just the main pawn next time.