Have you tried turning on forwarding and just letting the kernel take care of routing?
Code:
echo 1 > /proc/sys/net/ipv4/ip_forward
As long as the 2 bridges are configured in separate broadcast domains, this should work (ie. BR1=> 192.168.1.1/24 and BR4=> 192.168.4.1/24). You should see the two entries in the routing table....