Spooky scary skeletons!

Seriously, they are the best. Yes, they cost a lot of gold (cost increases after every purchase), but they are very efficient.
They can be used only as teamsters, but they do not rest nor require food. It is good to have them as workers in every vital building like Rations building, Smuggler, Lumberjack. Eventually I always aimed to replace most teamsters with skeletons.

Capturing prisoners is rather tricky. Farmsteads will give you (as you said), very few captives, however two or three is rather unlucky amount. Good idea is to get a small ship, and send it to explore nearby coasts. With a bit of luck you will discover Merchant Stations or Garrisons (I might get the names wrong). They provide more captives, and increased chance of specialists, but, especially Garrisons/Forts, are well protected, so having a Snow (cheapest ship) to raid it, is not a good idea.