I was going to include the main threads I used to get my game running, but apparently GOG has currently disabled links for posts... maybe I can find a workaround and add them in a separate post. (See the second post for the "links", you'll have to add the "www." yourself unfortunately... there's some good info in them, though.)
-----
There are a lot of different suggestions out there for getting this game to run. Some will work on certain systems, others won't. This is what I did to get the game to run on MY setup, hopefully some of the steps will help you progress towards being fully operational on your own rig.

I got my game running on Windows 7, 64-bit, with an AMD Radeon R9 200 Series graphics card.
You can check your own settings by going to your windows search under "Start" and typing in "dxdiag".
-----
Step 1 - Where you install the game matters.

To begin, as the most commonly advised practice from every help thread out there, don't use GOG Galaxy to install the game. Download the installer directly from GOG and use that instead.

According to one post from a user on Steam, this game in particular can have issues launching/running if the directory's length you install to is too many characters long (and, most likely, if it contains unorthodox characters such as [brackets]). I'm not entirely sure how the installer determines what folder it wants to install to, but wherever it chooses, check to see which directory it chose. I read somewhere that the game may have issues running off of drives other than your primary one, so you may want to experiment with running off of different drives as part of your troubleshooting process. However, moving your game directory around after installation can cause problems in the registry. To solve this, go to your windows search under "Start" and type "regedit". From there, navigate to the Rogue Squadron folder:

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\LucasArts Entertainment Company LLC\Rogue Squadron\v1.0\

(Ignore the "Wow6432Node" part if you're running a 32-bit OS.)

From here, right click on each of the files and make sure to replace the directory paths with the correct ones from wherever you decided to move the files. Be careful to include any executables at the end of each replacement, as some of the folders are more specific than others.

Playing around with the install location is what finally allowed me to see the menu screen after the intro screen (instead of just a black screen with music playing), which seems to be one of the most commonly encountered problems by users.
-----
Step 2 - Update your graphics card drivers.

Personally, I don't think this step helped me out too much, though it may have had an impact that I'm unaware of. Whatever the case, there are a lot of users that claim that updating their graphics card drivers helped solve the black menu screen issue. I would recommend giving this step a try, as updating your graphics card drivers is probably a good idea regardless of the impact it has on getting the game running.
-----
Step 3 - Install and configure nGlide.

For anyone that's tried running older games on newer systems before, this software should seem familiar. For new users, the setup can seem a little bit strange at first, but is simple to carry out.
Start by downloading nGlide:

(Couldn't include the DL link in this post, just Google it, I'm sure you'll find it as one of the first results.)

(Personally, I let it install in the default windows folder for simplicity, I recommend you do the same. You'll probably want to let it create a shortcut on your desktop as well.)

Next, you're going to need to go to your install folder for Rogue Squadron. Find "ROGUE.EXE", create a shortcut to it, and put it somewhere easy to access, like your desktop.

Right click on that shortcut and go to "Properties".

Where it reads "Target:", go to the end of the directory name, press the spacebar once, and add "-voodoo3" without the quotation marks.

Then hit "Apply".

Now when you double-click that shortcut, you should get a menu that has "Hardware Configuration" as an option. If it's not there, and all you can see is "Install Rogue Squadron", you may need to try launching the game a few more times using your original "ROGUE.EXE" and/or "Rogue Squadron.EXE" executable(s), to get it to recognize that the game is, in fact, installed.

Under "Hardware Configuration", click "Change 3D Video Card". Then make sure that "Voodoo Card (Driver: GLIDE)" is selected. Hit accept (if it says "If Glide cannot initialize, D3D will be used by default.", that's normal, just hit "OK".) and close out of that window.

Finally, you can change the settings for Glide by using the "nGlide Configurator" shortcut. However, take caution when changing these settings. For instance, I tried turning off V-Sync during my troubleshooting process. While this seemed to be an acceptable setting for going through the menus at first, I soon realized that disabling V-Sync caused the transition screens in the menus and cut-scenes to run incredibly slowly or freeze entirely. While it may help you to get your game running to play around with these settings, be careful that you don't accidentally forget about the settings and have them cause you even more problems later on!
-----
Step 4 - Running the game.

With any luck, you should have a game that's about ready for launch. There are a few more things you can check before starting, however. If you right click on "Rogue Squadron.EXE", and click on the "Compatibility" tab, you'll notice a series of checkboxes. First and foremost, I'd recommend going to the bottom-most checkbox, labelled "Run this program as an administrator", and turning it on.

Most people recommend disabling the other compatibility mode options (or leaving them disabled, as is most likely the case), which I'm inclined to agree with... however, it may be worth your while to experiment with some of the different OS compatibility modes if you're still having trouble launching the game.

Personally, I also disabled Aero and turned off my second monitor during testing, which might be things you want to consider trying as well. You never know what may be the turning-point with getting these older games to run.

(You'll have to add the "www." to each of these links unfortunately, since GOG seems to have links hard-disabled for new posters.)

Help threads I used to get the game running:

gog.com/forum/star_wars_rogue_squadron_3d/cant_access_the_game_menu_after_the_intro_black_screen/page1

gog.com/forum/star_wars_rogue_squadron_3d/resolution_changes_on_launch

gog.com/forum/star_wars_rogue_squadron_3d/keyboard_input_not_working

steamcommunity.com/app/455910/discussions/0/385429254952028569/

gog.com/forum/star_wars_rogue_squadron_3d/how_i_got_the_game_to_run_perfect

Download link for nGlide:

zeus-software.com/downloads/nglide

I followed your guide to the letter, and ... it worked! Until it didn't ....

Installed to the default location via the standard gog installer, moved the game to a simpler, shorter path with no spaces in it, adjusted all the registry entries to account for it, installed the latest nGlide software, ran the hardware setup (as administrator), and executed the game (as administrator).

And ... for the first time I got past the intro videos with no black screen! I was even able to click on the 'create' a player button ... and then the game crashed! The cursor was even still moving around the name the game generated for me, but ENTER and ESC did nothing and the game was ... dead. Had to use task manager to kill it!

I even uninstalled the game, deleted the registry, and reinstalled directly to the simpler path, with the same result.

After that process, here is what the Windows registry generated. Checking to see if this is all okay, or if maybe some of those directories should be changed:

---- (quote)

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\LucasArts Entertainment Company LLC]

[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\LucasArts Entertainment Company LLC\Rogue Squadron]

[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\LucasArts Entertainment Company LLC\Rogue Squadron\v1.0]
"Executable"="C:\\Games\\Rogue3D\\ROGUE\\Rogue Squadron.EXE"
"Install Path"="C:\\Games\\Rogue3D\\ROGUE"
"Source Path"="C:\\Games\\Rogue3D\\ROGUE"
"CD Path"="C:\\Games\\Rogue3D\\ROGUE"
"Analyze Path"="C:\\Games\\Rogue3D\\ROGUE\\SysCheck.exe"
"Source Dir"="C:\\Games\\Rogue3D\\"
"JoystickID"="1"
"SoundCard"="TRUE"
"InstallType"=dword:00000009
"Magic"=dword:0009283c
"DefVidID"="1"
"Driver"="GLIDE"
"VDEVICE"="Voodoo (Glide)"
"3DSetup"="TRUE"
"CurrentPlayer"=dword:000000ff

---- (unquote)

Still best to use dgVoodoo2 to run it in high resolutions unsupported by the game, or windowed mode, with flawless performance.
http://dege.freeweb.hu/dgVoodoo2/dgVoodoo2.html
If your antivirus blocks this exe it's a false positive, it is safe and very popular. My antivirus used to flag it but has fixed it since.

Anyway, all I needed to do was extract it all then plop the DLL files from dgVoodoo 2's MS folder into the same folder as the game's exe files are located and finally drop dgVoodooCpl.exe in there as well and simply run it to configure the game.

The settings I personally use are MAX (for me 1440p) resolution in the DirectX tab with the best virtual GPU - dgVoodoo Virtual 3D Accelerated Card with 2048MB VRAM - disabling the watermarks and enabling whatever else I like and you may or may not, like 16xAF and 8xAA. In the general tab make sure you choose the Centered scaling mode, for some reason forcing 4:3 or other modes doesn't work (in the latest version at the time of writing) but Centered somehow makes it so. This could differ in later versions after this post so that's the setting you need to fiddle with in order to get the proper aspect ratio (round radar in-game).

When you first run the game you might need to go to the "Hardware Configuration" in the launcher, then "Change 3D Video Card" and select the dgVoodoo DirectX Wrapper. Or the game might automatically do that perceiving it as the only option.

Absolutely nothing else was needed, no compatibility mode settings or tweaks or other programs or whatever else.

Information past this point is purely optional stuff, the instructions you need to get it running flawlessly are above this point.

The camera bug some people discuss where the view becomes way too distant as you play is maintained but I prefer the cockpit camera or the close-up camera right after the cockpit camera in the view cycle anyway so it's not a problem for me. Just make cycling the camera (or choosing one with the F keys) your first job in a level.

There is a way to use dgVoodoo 2 from a central location so you don't need to clone the exe in every game's folder (though you'll need to copy the relevant DLL files manually still) but you can figure that out yourselves if you want (or if you even have more games where it would work, currently I have Rogue Squadron 3D, MechWarrior 4: Mercenaries - which btw requires the VRAM to be set to 1024MB, it doesn't run with 2048 - and SWAT 3 installed so I've done that). This is just a short guide to play this game using simple steps anyone can manage if they know how to download and extract/move files.

All of the above on a Windows 10 64 bit installation with an i7 CPU and GTX1080 GPU so definitely very far from what the game was originally intended to run on back in 1998 and yet dgVoodoo 2 makes it work and perform like a charm.

Regarding the windowed mode I mentioned is possible, you leave all dgVoodoo2 settings as above, on the general tab select Windowed, on the DirectX tab disable the Application Controlled fullscreen/windowed state, set your desired resolution mode (on my 1440p desktop I used 1600x1200 which is 4:3 to ensure the aspect ratio isn't messed up, you might need to change more settings if you want a different size - btw the game by default has a 1280x1024 option but if you use that it actually squishes the hud a bit, t's not standard 4:3!) and the game should go windowed after several seconds of the intro play (for some reason that video always starts fullscreen and is also stretched rather than maintain the aspect ratio as it does when you have it in fullscreen mode with the same settings). If it doesn't you can always try alt + enter to force it after you're in the main menu (make sure the Disable Alt+Enter Toggle in the DirectX tab isn't enabled for this shortcut to work at all).

The mouse controls are weird past that point because the capture mouse setting in the general tab doesn't appear to work so the in game cursor is in a different position than your mouse cursor, you juggle it or end up clicking out of the game and unfocusing. The gamepad (joystick, whatever) controls remain intact so it's fine. You can also enable the center app window on the general tab if you want the game window to not be on the desktop's top left. I tried moving it around with the mouse but it just resized (maybe it's possible if you fiddle around, try and see, don't worry about the size as the next time you start it, it will resize to the chosen resolution still). Keep in mind the game has to be in focus otherwise it auto pauses and some functions (like windows key + print screen to save a screenshot) don't seem to work (thankfully the MSI Afterburner screenshot shortcut works).