I have this same exact problem and I am about ready to give up on both these game. Also, the screen seems to randomly shake at times making it extremely hard to aim. Is this just how the game is with mouselook or is it a result of the dosbox, mouse aim something else?
Also is Descent 3 the same way or is it smoother? I will probably pass on it if it is the same.
You must remember that D1 and D2 both are extremely old games and tend to have issues with control. You have two routes that should solve your problems: D2X-XL and DXX-Rebirth.
D2X-XL (Thanks to Diedle, AKA Karx11erx) has significant changes to the original Descent engine that allow it to run with high-res textures, models, levels, and includes numerous lighting effects, smoke effects, shadow effects, etc. - basically, if the geometry were not so simplistic, theoretically it could look as good or better than Doom 3. Control options are in plenty here; they should be able to fix the mouse issues. This should also fix the shaking issues, because if I recall correctly they are caused by excessive framerate: D1 or D2 straight out-of-the-box will have framerates it was never meant to have, thus leading to some shaking issues (and the infamous homing missiles from D1 that on newer machines are completely unavoidable). Note that in D2X-XL, all files must be rearranged into different folders, which can be a bit time consuming.
DXX-Rebirth (Thanks to Christian Beckhaeuser, AKA Zico) is quite a bit more spartan in it's approach. It's aim is to make a faithful, ultra-stable application for D1 and D2. Few features exist, but it is the fastest and easiest way to experience Descent on modern hardware properly. These should also have the controls to fix the mouse issues, and if shaking still occurs, try setting the game to a higher resolution or adding a framecap in the .ini files (explained better in the readme).
Take your pick. I prefer DXX-Rebirth due to it's simplicity and ease (sometimes, less is more, IMHO), but for the advanced user or someone who really wants to push the limits of Descent and see it in hi-res, hi-poly glory, D2X-XL is the way to go. =)