Exactly what it says on the tin. :)
There are numerous options, including Overlay, Surface, Direct3D, DDraw and OpenGL.
I'm wondering if there's any difference between them. Also I like to run my games through Steam, but only some renderers will allow you to take use of the Steam Overlay - wondering if the selectable graphic modes will have an impact on that?

I would not bother with any renderers other than openGL and Direct3D if the Steam Overlay is something you want to use while playing Tyrian.

As a rule it's only worth tinkering with those if you're having compatibility issues. But there is one trick worth knowing if you're using a widescreen monitor which involves that DOSBox setting. In the *.conf file the game uses, change the following lines to these values:

fullscreen=true
fulldouble=true
fullresolution=0x0
output=ddraw
aspect=false

That'll often make sure the game fills the whole of the screen without causing too much aspect ratio distortion (it does with Tyrian, for example). :)

The reasons being that the original resolution of the game was 320x200 (try pressing Ctrl+F5 during play to get DOSBox to take a screenshot and then check its dimensions). I'm using a widescreen monitor that I run at either 1280x768 or 1600x1024. So now compare the ratios:

320 / 200 = 1.6
1280 / 768 = 1.666666...
1600 / 1024 = 1.5625

All fairly close. Whereas, if you try to run it in a 4:3 ratio like 1024x768...

1024 / 768 = 1.333333

All this babbling gibberish means that for many old games that ran at 320x200, using the above trick on your widescreen is a quick and easy way of stretching the image to fill the whole screen without aspect distortion. :)