Have you tried running the game in 32 bit mode and disabling one of your cores for it?

You have to run as administrator. Right click on the shortcut and click "run as administrator".
Not sure why it's so. I think it's because the program needs to write files in the "program files" directory and Vista doesn't allow that for people logged in under a standard account. Any techies want to answer this?

Running as an administrator is not needed for this and is not the source of the problem. Duke Nukem 3D is a DOS game; 64-bit versions of Windows do not support 16-bit software (which is anything made for DOS or Windows 3.x). You cannot run it from the DOS EXE itself; you must run the game through DOSBox or a DOSBox shortcut in order to play. The GOG installer will have created Start Menu shortcuts already set up to do this, or you can use your own DOSBox installation.

Also, here's my preferred settings for the game:
1) Once you're in a game hit "u" to turn on mouse look. Hit "i" to turn on crosshairs.
2) Exit the game. Start the "setup" program and go to controller setup --> setup advanced control options --> advanced mouse setup. Highlight "mouse y axis" and hit delete. Save and exit.
You can bind keyboard shortcuts in the controller setup --> setup keyboard menu. Use the delete key to get rid of any bindings that are duplicates or ones you don't want.
Some people will recommend eDuke. That works but I prefer to keep the experience as authentic as possible. :)
Arkose,
I am running Vista 64. I run the game by clicking on the shortcut that was created in the start menu which is really just a shortcut to dosbox. I had the same problem the OP had with the window closing while the game was starting up. Running as administrator works, however...

Thanks blsessex, the 'Run As Administrator' tip worked for me (I had the same problem as OP), and I'm running Vista 32 bit.