I've figured out that it's because my gun is out, but is there any way to get them to stop without killing everyone?

Yes. Put your gun away.

If you don't want to walk unarmed, equip your gun in the second slot. That way if you have to defend yourself, you don't have to waste 4 action points to go to your inventory and picking the weapon. It doesn't matter if you have a bazooka in your other hand, they don't care, as long as you're not shaking it at their face. :p