First download the patcher for the executable here:

http://www.vogons.org/viewtopic.php?f=24&t=41331

Apply the patcher to the BoP executable.

NOTE: the archive on that forum thread also contains patchers for the XW95 and TIE95 executables, I never tried them as I still have my DOS cd-roms but they should work in much the same way.

Now you should be able to go to the game options menu and enable 3D acceleration but with recent video cards it usually refuses to work and goes back on software mode when loading a mission, for this we need the dgVoodoo2 wrapper to emulate an older video card (not necessarily a Voodoo2 as the Direct3D wrapper also supports old GeForce and Radeon).

You can download the latest version of dgVoodoo2 from this page:

http://dege.freeweb.hu/dgVoodoo2.html

Open the archive, copy the DLLs inside the "MS" folder in the BoP executable folder (note: if there are already DLLs with the same name make a backup).

Now open dgVoodoo2 to create a config file that will have to be put in the same folder as the BoP executable.

Experiment a bit with the settings (the ones that matter for BoP are in the "general" and "directX" tabs), I was able to force a higher resolution (1280x1024), trilinear filtering (by forcing both bilinear and linear), Vsync and 8X antialiasing without issues but that might not be the case for other video cards (mine is a Radeon HD 7770 1GB).

Click on "apply" to generate a config file and copy it in the same folder as the DLLs and the BoP executable, now 3D acceleration should be working with all the options you selected.

It would be great, if the gog team would integrate such things. Without 3D XvT looks horrible.

No really, it didn't use 3D acceleration that much apart from bilinear filtering for the textures, there wasn't even an effect for beam weapons like in X-wing Alliance and screen resolution was the same as the DOS Tie Fighter CD-rom, even the poly count looks largely the same as TF only with texture mapping and a few light effects that were there even in software mode.
Now I'm concentrating on my "normal" playthrough of Tie Fighter (DOS CD-rom) and trying also to complete Wing Commander Saga (the fan game) so I couldn't test it thoroughly and there might be issues that I just didn't find yet.

this worked for me great;

I would add one more thing, given that when I made the above changes the mouse cursor became invisible!!!

the fix is to add a '-softwarecursor' handle to the start up shortcut, and the cursor should be visible again...

I should probably edit the post, now I use a d3d11 wrapper specifically made for XvT and XWA which is more straightforward and does a better work of upscaling the 2D elements when upping the 3D resolution.
https://github.com/rdoeffinger/xwa_ddraw_d3d11/releases/latest

Unless there's a specific reason, on any GitHub always link the binaries with a "/release/latest/" URL; so thay it will always redirect to the more recent one... 1.5.9 is already being replaced

https://github.com/rdoeffinger/xwa_ddraw_d3d11/releases/latest

Thanks!