One of the things I can think of is that, since you were already a vessel of the Changing God, and normally the Changing God would transfer into another vessel (and not just hide within the mind maze thing), that he probably thought it wouldn't make sense to then re-enter the discarded vessel?
Then why did he change his mind later on? Besides, from what I understand, it was not his choice to enter the Labyrinth. One of the last meres tells us that the real Changing God was killed by the Sorrow during his escape from the space station. Probably he didn't manage to transfer his consciousness to another body in time. It's controversial whether we view the Phantom as a real Changing God. The same mere tells us that the Phantom was only his memory backup made by himself before the fall. But again, it depends on how you look at it. If something is a faithful copy of consciousness, can we recognize it as the original? The Phantom undoubtedly had all experiences and personality of the Changing God, but is the copy equal to the original? We can convince the Phantom that he's not the real Changing God before entering the Resonance. Depending on your point of view, you can treat him as the Changing God or assume that the real Changing God died. Anyway, I believe it was not his will the get stuck in the Labyrinth.
Also there's the plan about merging all the castoffs - maybe he thought this plan was so close to fruition that it's somehow better to hide in the mind maze until the Last Castoff joins everyone? Although this kind of makes less sense - one would think that he'd want to re-enter the Last Castoff to ensure that his plan works perfectly, instead of letting him fumble around...
The only thing he needed to do is to repair the Resonance Chamber. It would be more efficient, if he decided to do this by himself. He had all the proper wisdom to repair it. What more, to use the Resonance Chamber he needed Last Castoff's body as a catalyst and what's the better way to ensure that the body will be intact than to take it over? So this is not a good explanation in my opinion as well.
Which leaves the only other thing I can think of - that maybe the Phantom wasn't entirely faking his memory loss when he first meets you, and that maybe it's not until Miel Avest that he finally thinks that he's The Changing God, and has to take control?
I think this is the best explanation. The fall was preceded by very intense events that could contribute to the memory loss. And when the Last Castoff's consciousness took over the body, it has probably made havoc to the mind deepening the memory problems. Falling to the ground was also a shock, which could have influence on this state. Seems reasonable to me.