At the moment (normal difficulty) I try a game without walls. You get an early message about the raiders and if you mark immediatly all people with "I" and sent them towards the raiders the settlers easily win. You lose two or three people but that doesn't matter. Even later with bigger raiding parties simply sent all villagers with weapon against them. No problem. 'Til the next raider atttack your loss is more than replenished.
I successfully used approach of building just gates and 4 towers per gate for several games. When you sound alarm horn, people run home and equip weapons and go to the gate where enemies are expected to strike. And not only people, but also dogs, which is important as sometimes I survive without human casualties, only few dead dogs.
There I select full group of warriors with "i" and when the first attacker goes for tower, I send whole group at him.
Towers function as a distraction / temporary damage soak. Usually there are some casualties on my side, but less than there would be without towers. And towers are easily rebuilt, while it takes time to obtain more people.