I just figured it out. The patch made it unplayable. I reinstalled it without the patch and now it works fine.
Thanks man, you just saved me a whole lot of trouble. Had the same problem with the game crashing on startup and I already thought it's because I have Intel HD graphics but then I tried installing it without the patch and that works perfectly fine - except that in the unpatched release, there seems to be no gamepad support. I just bought an XBOX controller today and wanted to play The Swapper with that. Hmph :(
(The controller works fine in other games, Stick it to the Man for example)