Unless you're blasting off spells that require lots of gems left and right you'll find yourself with a serious surplus of gems as the game progresses (especially in Darkside of Xeen), so the measly amount of gems required for the stat increases in the Cave of Illusion is well worth it.