In the Mouse-Joystick control options in the controls settings you probably have the mouse control mode set to "mouselook" instead of "flight sim". Turn it to "flight sim" to fix the janky mouse.


Ordinarily the ship in the Descent series has a maximum turn speed. But, this made some mouse users uncomfortable because when they try to do a snap 180 turn like in Counterstrike the ship would barely move and they decided the mouse in Descent must be broken.

Thus Descent 3 added an option for mouse users to break the turn speed limit. Unfortunately it's poorly implemented and sloppy as you have noticed. Therefore the original "flight sim" limited turn speed mode feels much smoother. Just remember that only slower mouse movements will register, if you move the mouse faster than the ship turns the movement will be wasted. But, D3's ship turns a lot faster than D1/D2 so it won't be as noticeable.