I've had to abandon this fight every time I've met him. If you fail to stun him early, once he uses Collect Call I can't touch him again: the minions intercept any attack directed at him & once you get through a minion he just calls more.

He seems to have low move resistance but I've never been able to move him into a position where the minions don't protect him - or I run into him when I don't have a move hero in the party.

Stun breaks guard? How did I not already know that? Thanks!

I said about stuns and The Collector here: https://www.gog.com/forum/darkest_dungeon/bosses_champion_darkest

PS. If The Collector is guarded you have two options with the same effect in the end:
1) directly stun Collected Man-at-Arms
2) try to stun The Collector - stun will be redirected to Collected Man-at-Arms

I realize you have already gotten a good answer, but I thought I'd add one more strategy that has helped me at times in dealing with enemies under guard. Use the Damage over Time/Area Effect attacks that hit more than 1 enemy at a time and continually bleed/blight the enemy. For instance, the Plague Doctor's plague grenade (the stun grenade is also good for this) will hit both back ranks of the enemy, so if the guarding enemy is 3rd and Collector 4th, you'll hit them both. Hound's harry can also do this since it can bleed all 4 ranks (use the dog treat first). Use push/pull attacks (Occultist demon pull or Bounty Hunter uppercut for instance) to position the guard as needed. And don't forget that all bleed/blight stuff stacks, so the more you add on, the quicker the guard and/or collector goes down. As I understand it, you are playing the base game, but another way to combat this is with the Shieldbreaker hero in the DLC as she has specific moves to counter enemy guarding. Stunning is still probably the best way though.

A few other points about the collector fight:

* He has a relatively low speed and manageable stun resistance, so opening with a stun and dealing almost two rounds worth of damage to him before he even uses collect call is eminently possible. I have killed him more than once before he even uses Collect Call.
* There's a fair bit of luck in the collected enemies he summons and what their AI chooses to do early on. Worst combination is probably Highwayman/Highwayman/MaA. Sometimes the MaA guards a highwayman instead of the collector, which is helpful. Vestals can generally be ignored as they often heal the wrong targets and you can out-damage them anyway.
* I would always try to focus the Collector down first; he's not that tough or hard to kill and so long as he's alive he keeps summoning more. Then take out the highwaymen next, obviously.

If you have a party with no stuns, you are in a lot of trouble unless you get lucky. Basically, make sure parties do have reasonable access to stuns.