setup.exe is supposed to be run in dosbox, not native windows.
i don't have the gog version, i bought my Duke CD a long, long time ago and I use eduke32. I mention this so my next statement doesn't seem as stupid as usuall.
Isn't there a dosbox shortcut created to run the setup.exe, and not just the duke3d.exe, when the game is installed?
make a copy of the dosbox config file that runs duke3d.exe. edit it with notepad and change all instances of duke3d.exe with setup.exe.