I got caught stealing in Gargan, the Shipwrights's house in Trinsic and now he keeps following me around town. I tried to put the item back in the chest, where I found it and tried to hand it over to Gargan. Nothing seems to work and since he's following me I can't steal anything else in Trinsic without him alerting the guards.
Does anyone know how to solve this?

You should be able to take anything without being caught as long as the game is paused. Companions will complain and eventually leave if you put them into your own bag, but not if it's one of their bags.

In Serpent Isle it's a bit different. You need to break line of sight to any witnesses, and companions won't care no matter who grabs what.

Thanks for your help, @Gotcha and @GeistSR.