I really hope you already got in contact with gog support. My guess is that Boxer has a problem to recognize your joystick correctly. Unfortunately GOG uses the standalone version of Boxer which has no option to configure the joystick, in that version Boxer automatically choose a joystick preset. Only the normal version of Boxer has also some options to change joystick presets.
What you can try now is to run the game by using the original Boxer-version, which is kind of tricky. To do so download and drop the Boxer.app in your program folder. Then right-click on gog X-Wing app and choose "show package contents". Browse to contents/Resources/game and do the same with the file located there (right click and show package contents). Now browse to contents/resources and you will find a file called "Star Wars™- X-Wing Special Edition.boxer". That's the game package. Double-click on it and it should start with the normal Boxer version. You will recognize that there are 3 new buttons left on the status bar. In case you play the Special CD-Edition you will also get the dos-prompt and an error message that the game can't find the cd. Just choose and click on start.bat from the progam launcher Boxer offers you at the bottom (you need to scroll to the right), the game should start now. To lock out your mouse to reach the status bar just press the command-key and then press on the left button with the magnifer-symbol, you now should get the advanced control options including the joystick presets. Try to choose another joystick preset and see how it works. To get in fullscreen, press command+F. If everything works for you, you can copy "Star Wars™- X-Wing Special Edition.boxer" also to another folder (as example the desktop). After starting the game by choosing start.bat just press the middle button and say Boxer to always choose start.bat, next time you click on that file the game should start out of the box.
Again, that's just a dirty workaround, you really should contact support. As a sidenote, I tried the game with my Logitech RumblePad 2 and did a very short test flight. It seems to work ok for a gamepad.
Boxer homepage
http://boxerapp.com/
Boxer and joystick-support
http://www.gog.com/forum/star_wars_tie_fighter_special_edition/mac_how_to_disable_automatic_joystick_button_mapping/post2