If your toolchain is built with multilib support, then you can do something like this when you configure the package. You need to have pkg-config installed.
CC="gcc -m32" CXX="g++ -m32" \
Take a look at the multilib build instructions at CBLFS
to get a better idea.