If your toolchain is built with multilib support, then you can do something like this when you configure the package. You need to have pkg-config installed.
Code:
CC="gcc -m32" CXX="g++ -m32" \
PKG_CONFIG_PATH="/usr/lib/pkgconfig:/usr/local/lib/pkgconfig:/opt/lib/pkgconfig" \
./configure
Take a look at the multilib build instructions at
CBLFS to get a better idea.