FYI - Hardware details - This is a Dell R610
- (4) ethernet ports
(2) - ports for cross over cables beween units - for 2gb transfer
(10 - DRAC port for remote management

NO ports are cabled except the (4) GB ethernet ports.

As long as I understand only 4 interfaces play:

bond0:
eth0 Link encap:Ethernet HWaddr 00:21:9B:9B:4F:21
eth1 Link encap:Ethernet HWaddr 00:21:9B:9B:4F:21

bond1:
eth2 Link encap:Ethernet HWaddr 00:21:9B:9B:4F:25
eth3 Link encap:Ethernet HWaddr 00:21:9B:9B:4F:25

bond0 Link encap:Ethernet HWaddr 00:21:9B:9B:4F:21
inet addr:10.35.13.85 Bcast:10.35.13.255 Mask:255.255.255.0
inet6 addr: fe80::221:9bff:fe9b:4f21/64 Scope:Link
RX bytes:147375 (143.9 KiB) TX bytes:158673 (154.9 KiB)

bond1 Link encap:Ethernet HWaddr 00:21:9B:9B:4F:25
inet addr:10.35.13.86 Bcast:10.35.13.255 Mask:255.255.255.0
inet6 addr: fe80::221:9bff:fe9b:4f25/64 Scope:Link
RX bytes:100805 (98.4 KiB) TX bytes:3744 (3.6 KiB)

It looks like bond1 - doesn't work at all: TX bytes:3744 (3.6 KiB)
May you assign all ethernet cards to bond0?

in your ifconfig -a listing :

eth0 and eth1 have the same hardware address
eth2 and eth3 have the same hardware address
eth4 and eth5 have different hardware addresses

in the cat output

ifcfg-eth0 hw address matches the ifconfig -a output
ifcfg-eth1 hw address does not match the ifconfig -a output
eth2 hw matches
eth3 hw does not match
eth4 hw matches
eth5 hw matches

so ifcfg-eth0, ifcfg-eth2, ifcfg-eth4 and ifcfg-eth5 all match up with the output of ifconfig -a
ifcfg-eth1 and ifcfg-eth3 are bogus entries.

#########################
IMPORTANT I did not know about the other interfaces when I wrote the stuff below.
You may just need to make sure the contents of the ifcfg-ethx files match the correct addresses.
#########################
#Delete those 2 files. (or rename them preferably)
#Delete all the backup files (ifcfg-ethX~)
#rename the others according to which real address they have according to #ifconfig and make sure that you have two slaving to bond0 and two slaving to #bond1 (check the MASTER lines)
#then restart the network.
############################
Our posts crossed over, I didn't know about the other two ports and the DRAC port. Do you know what their hardware addresses are ?

Sorry now I'm concerned.
- Before I delete any files, (which I'm not sure is right) since if the machine has (4) ethernet ports, (2) 2gb crossover machine data ports and (1) DRAC
port. I believe the (4) ethernet ports are eth0, eth1, eth2, eth3 (I'm not sure what eth4 and eth5 are but they are not up)
- I'm not sure how the eth0, eth1, eth2, eth3 files were populated with the HW addresses - I suspect he retreived them from the GNOME network utility,
not sure
- What command or where CAN you get the real MAC addresses of eth0, eth1, eth2, eth3 ?
- I agree with what you stated matches ifconfig and the cat of the ifcfg-ethx files.
- Actually I'm ready to bond all (4) interfaces to bond0 tomorrow since I spent the entire day trying to figure this mess out.
- so if I move to bond0 only, I just need the specific mac addresses of eht0, eth1, eth2, eth3 so I can make sure their right in the ifcfg-ethx files.
How can I make sure I have the right MAC addresses to populate the files since I think it's best given the architecture and project to move to
(4) port bond0 instead of (2)x(2) bonding failover...which is a mess.

Thank you

Command to see the real ethernet configuration is: "ifconfig ethN" without "HWADDR=" in ifcfg-eth. I do not know why you changed them, because looks like they become equal to their bond MAC address.
Also please remember, that "ifconfig" without any additional arguments, shows ACTIVE interfaces.
With "-a" all of them.

I need the true MAC addresses of eth0, eth1, eth2 and eth3 the 4 gb nics.
ifcfg eth0..4 shows duplicate MACs
[root@hhlxnrcsas02 ~]# ifconfig eth0
eth0 Link encap:Ethernet HWaddr 00:21:9B:9B:4F:21
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:12517 errors:0 dropped:0 overruns:0 frame:0
TX packets:477 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3112291 (2.9 MiB) TX bytes:80601 (78.7 KiB)
Interrupt:36 Memory:d2000000-d2012800

[root@hhlxnrcsas02 ~]# ifconfig eth1
eth1 Link encap:Ethernet HWaddr 00:21:9B:9B:4F:21
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:446 errors:0 dropped:0 overruns:0 frame:0
TX packets:492 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:56300 (54.9 KiB) TX bytes:89239 (87.1 KiB)
Interrupt:48 Memory:d4000000-d4012800

[root@hhlxnrcsas02 ~]# ifconfig eth2
eth2 Link encap:Ethernet HWaddr 00:21:9B:9B:4F:25
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:11826 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3030807 (2.8 MiB) TX bytes:1798 (1.7 KiB)
Interrupt:32 Memory:d6000000-d6012800

[root@hhlxnrcsas02 ~]# ifconfig eth3
eth3 Link encap:Ethernet HWaddr 00:21:9B:9B:4F:25
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:446 errors:0 dropped:0 overruns:0 frame:0
TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:54896 (53.6 KiB) TX bytes:1852 (1.8 KiB)
Interrupt:42 Memory:d8000000-d8012800

[root@hhlxnrcsas02 ~]# ifconfig eth4
eth4 Link encap:Ethernet HWaddr 00:10:18:64:35:68
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Interrupt:40 Memory:da000000-da012800

[root@hhlxnrcsas02 ~]# ifconfig eth5
eth5 Link encap:Ethernet HWaddr 00:10:18:64:35:6A
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Interrupt:50 Memory:dc000000-dc012800

[root@hhlxnrcsas02 ~]#

Because you bond them together, I think:

# Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet
DEVICE=eth0
HWADDR=00:21:9B:9B:4F:21
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=no
[root@hhlxnrcsas02 network-scripts]# cat ifcfg-eth1
# Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet
DEVICE=eth1
HWADDR=00:21:9B:9B:4F:23
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=no

Leave only "DEVICE=eth1" in "ifcfg-eth1" and execute ifup eth1, then check with "ifconfig eth1"

And one MAC + one IP for one interface bonding sounds very reasonably, because if one on other side decide to implement mac filtering all your connection from all interfaces in one bonding will be with one IP and MAC.

I agree with nimnull22.

We are only seeing the results of the bonding in ifconfig -a
this is how the fedora docs say a ifcfg-eth<N> file should look :
This is how they say the bonding file ifcfg-bond0 should look
I would copy this format (notice no mention of hardware addresses) using the correct details for bonding mode, ip address and network into the relevant files.
Your ifcg-bond0.conf and ifcfg-bond1.conf files are mostly right already, just uncomment the options line in each.

Then comment out the options line in your bond0.conf and bond1.conf files found in modprobe.d
so that they just read

alias bond0 bonding

and

alias bond1 bonding


At least then you will be following the documented approach.
I think the hardware addresses were confusing, especially when we learned of the other 3 interfaces.
All the ifcfg-ethX.conf files should have ONBOOT=YES

I'll try this tomorrow and let you know what works.
I don't want to loose the connection and the current state.

Resolved !

The problems was as follows:
- the network guy never setup the ports for bonding and we had wired:
eth0 - sw1
eth1 - sw2
eth2 - sw1
eth3 - sw2

We changed the bonding to bond eth0+eth2 and eth1+eth3 and it worked fine.

Thanks to all...