I would like to improve the throughput of my db servers, running CentOS4.7 64bit. I wanted to bond the 2 interfaces together, and active active configuration to provide more throughput and provide fault tolerance. My question is this, my network is a flat Layer 2, with 2 gigabit dumb switches, so no Spanning Tree Protocol (STP). If I bond the 2 nics active active, then plug one nic in one switch and one nic in the other switch, would I create a routing loop?

no you dont need it, Here are my config files you can copy them to the /etc/sysconfig/network-scripts/

ifcfg-bond0
----------------------------------------
DEVICE=bond0
BOOTPROTO=none
IPADDR=10.XX.XX.1
NETMASK=255.255.255.248
NETWORK=10.12.XX.XX
BROADCAST=10.12.XX.XX
ONBOOT=yes
USERCTL=no
----------------------------------------
ifcfg-eth0
----------------------------------------
# Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet
DEVICE=eth0
MASTER=bond0
SLAVE=yes
HWADDR=00XXXXXXXXXXXX
BOOTPROTO=none
IPADDR=10.12.XX.XX
NETMASK=255.255.255.248
NETWORK=10.12.XX.XX
BROADCAST=10.12.XX.XX
ONBOOT=yes
USERCTL=no
----------------------------------------
also you need /etc/modprobe.conf
alias bond0 bonding
options bond0 miimon=100 mode=0
alias eth0 bnx2
alias eth1 bnx2