I believe Ultima Underworld had the first real 3D game engine ...
From what I recall, that's not quote
accurate. It can be argued to be the first texture-mapped
3D game engine, but untextured engines existed before it.
My issue is with gameplay design decisions surrounding the use of magic and missle weapons. Not with the game engine.
I think that what might be being suggested--and what I would suggest myself--is that it's a game old enough that developers hadn't yet learned how to (relatively) intuitively handle some aspects of interaction in a real-time 3D world. Indeed, I think that it's old enough that a number of game design matters that are now fairly well-settled were not yet so. (See mouselook, for example.)
As a result, it can indeed be janky.