Hey Ace. Sorry to hear that it's not working for you. It sounds like this game has issues for a lot of people. I downloaded the installer just a few minutes ago to see if it worked on my Win 7 x64. It did run for me, but I had a few issues.
On my system, I just downloaded the installer from galaxy, and then used the shortcut the installer provided. I didn't need to change anything in config or manually set any compatibility modes (the GOG installer set F18start.exe to WinXpSP2 compatibility/disable desktop composition/run as administrator by default).
The game did crash once on the "digital innovations" title screen,it seemed pushing too many keys on this screen led to a crash. Occasionally I would get a black screen with music playing in the background when selecting menu options. When I got the black screen, I just had to Alt+tab away from the game and back. Still, I was able to play both campaign missions and quickstart missions (some people in the forum have been unable to do campaign missions, but CAN play quickstart missions). I skipped all of the cinematics, I'm not sure if that made a difference or not.
I'm sorry I can't offer better advice. I assume you only set the config and compatibility modes after the game didn't work? If not, I would suggest an uninstall/reinstall with default settings.
There is another forum thread with possible solutions here: https://www.gog.com/forum/f_a_18e_super_hornet/just_another_this_game_doesnt_work_at_all_thread
maybe some of these suggestions would help? When you try to play, are you able to even get to the main menu, or do you get a black screen, or does the exe run but nothing happens, etc? Maybe knowing how the game fails will allow others to provide a solution to your problem?
If all else fails and you still can't get it to run, don't forget about GOG's refund policy. Again, this seems like one of those games that causes lots of problems on lots of different hardware. https://support.gog.com/hc/en-us/articles/115000487189-Money-Back-Guarantee-Policy
I hope some of this helps.