Here's one setting I changed that seemed to help a lot.
Just for reference, I originally tried playing on a Thrustmaster HOTAS X, but the massive dead zone rendered pretty much any game running in DOSBOX unplayable. On a whim, I tried playing with a wired XBOX 360 controller, and was surprised to find that there is little or no dead zone when using it. This suggested to me that the dead zone issue was (partially at least) due to the stick I was using.
Upgrading to a Thrustmaster T.16000M FCS HOTAS improved things greatly for some games (X-Wing and Descent) but did little for others (all the Wing Commander games).
I then tried changing some DOSBOX settings:
Joystick type: FCS
Timed interval: No
I don't think this improves the dead zone, but I think the timed interval setting is required to prevent drifting.
I then set
Emulation core: Normal
CPU Type: 386
CPU cycles: 3500
In my case this DRASTICALLY improved the dead zone for WC1 and 2. It's still there, but instead of covering a full 20-25% of the joystick range, the dead zone now only occupies the initial 5-10% of the range. It's not perfect, but it is very playable.
Finally, I cleared the keyboard mapper and left it off. I found that setting this ended up causing way more problems than it was worth. Better to just play these games the way they were intended, with just a stick, two buttons, and your keyboard for everything else.
Note that WC1 and 2 are not meant to play "smoothly." You can increase cycles in DOSBOX and get the game running extremely smooth, but it messes up all sorts of things. For example, the joystick becomes less responsive (larger dead zone) the faster you force the game to run (actually, my theory the joystick response stays the same, but the game is running smoother so it makes it much more noticeable). I would also encounter all sorts of game balance issues, like hyper aggressive enemies that could take me down in just a couple seconds. My theory is that a lot more than the frame rate is tied to the CPU speed.
Back in the day, playing these on original hardware in DOS, they were not 60 fps by any means. I clearly remember the games being a bit jumpy on my 386 (which was more than powerful enough to run the game). This means that the game will be quite smooth when nothing is going on, but will drop frames during hectic dogfights or when flying through asteroid belts. That's just how the game was back then.
Wing Commander Privateer is a different matter altogether however. I am tempted to just play that one with a mouse.
This also helped me significantly.