My very first POI that I encountered was the bandit camp on the cliff. I rolled away from them clear off to my death.
LOL. I did exactly the same thing.
But - I don't find the controls laggy at all. Very smooth and responsive for me...