Magnus joins the party on a specific condition (besides a free follower slot, positive alignment, being level 7 or above, and being capable of normal speech):
You have to sort out the deal with the P. Schuyler & Sons by killing the Schuylers. This is best done by recruiting Magnus before you go inside, since he is interested in them as well, and then he just stays with you if you let him. If you deal with them peacefully and then ask him to join, he'll refuse.
However, since he won't even talk to you, it's possible you've done something in an odd order and blocked the required triggers completely, normally he should at least have some response about not wanting to join you.
I know from experience that Magnus's party checks are a little off even with the unofficial patch, he once joined me as a free follower, which is very out of character for him. :) So it's very likely a glitch, and if so, there's nothing you can do about it besides reloading.