The other half of your party also has to fight, though that one is significantly easier. Keep your main tank with your character as well as your primary damage dealer. The gnome talks, you still have things to do - rebuff and extended haste and then talk to him and go.
Other side is a Venus ffly trap that can be kited.
I did it!! Did as you said, though I didn't know you don't have to divide the party equally, so I stayed with everyone and only sent Nok-Nok for the flytrap. Used some scorching rays and acid arrows.
Just hope the damn flower never comes back!
Thanks for the help!