Devil_Poul: You need to pay attention to population stats.
If the Workload is too high, for me above 85% is getting high and above 100% is risking many problems, workers will get tire much faster as their Stamina drops too low.
Many thing are important, like Morale and Hydration or Temperature.
Morale can be boosted by Spiritual and Burial constructions. Hydration need steady supply of water, be it gathered from river, lake or well and stored in storagehut or houses.
Temperature deppends on the map and is regulated by clothing. for ex.: On colder maps it is important that population has sepply of warm clothing for all the time.
Any, and all, of that stats getting too low results in population "sleepwalking", also during raids when You rise the alarm.
Don't know what is going with Your villagers, but try to investigate them, red markers above them indicates that they are in critical need of something.
They don't normally sleepwalk. But when the alarm is sounded, they start moving like snails. Multiple times slower than normally. And having enough warm clothes is my goal all the time.
