How do you build it usually?
I always compile the 64bit on host OS and the 32bit part in a VM, then merge the required files together.
EDIT: I play a lot of 32bit games, so that component is a must for me.
Out of curiosity, what specs do you use for the VM?
(Also, assuming your kernel can run 32-bit binaries, you could also use a chroot or container for the compiling, avoiding the overhead of a VM. And if your kernel can't run 32-bit binaries, why are you compiling a program that your system can't execute? (Note that not being able to run 32-bit binaries would be highly unusual; it probably means you compiled your own kernel and chose to disable 32-bit support, or you're working with some embedded distribution rather than one meant for desktop/laptop use.))
But I'll probably eventually set up 32-bit VM as well, since using faudio with staging would be better. Can be useful for such weird 32-bit outliers.
Again, you can do this with a chroot or a container and avoid the overhead of a VM.
(Quick way to get a debian chroot: use debootstrap. You may need to copy or hardlink resolv.conf into the chroot if you need network access (in particular, DNS) in it, but otherwise you should be fine for this purpose. Other distributions have similar tools; if you want an Arch or Gentoo chroot, the installation guide will have you setting up a chroot as part of the installation, so you could just follow that part of the guide.)