This isn't really a problem, though I am conflicted about it...

I played through about half the game to get myself reacclimated with things, before I decided to restart and plan my character. When I did, I realized I still had all my fancy gear (Dragon's Tooth Sword, lots of multitools and lockpicks, fully modded guns, etc) right at the start on liberty island!

I'm a bit conflicted about this. I wanna keep my sword but it's totally out of place. I'll probably just chuck most of it in the water. Just thought this was an interesting glitch I'd share.

As i played the game i restarted several times, trying out new char-settings and choosing a different ending.
The Bug you described never appeared for me, did you try to reproduce it?

I believe this glitch happens when you are in the middle of a game, and you start a new one without quitting to the main menu. You will keep your entire inventory from the previous game.