NIC PCI Addresses

hybernate20 · 08-26-2009, 02:58 PM

Hello,

Hopefully this makes enough sense...

When setting up servers, I have to bond our network interfaces. We use SuSE Enterprise, and since it sucks at bonding with Yast I just do it manually. This requires that I know which hardware address goes to which nic (which we generally have at least eth0-5). I can do an lspci | grep Ethernet and come up with a list of addresses and which NIC is detected... but when all of the cards are the same model it makes it kind of hard. I have to usually enslave the wrong interfaces at least once before I can figure out which NIC goes to which address.

My question is this: is there a good way to know which pci address goes to which nic specifically without trial and error? I can run an lspci | grep Ethernet and get this:

03:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
05:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
0b:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
0b:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
14:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
14:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)

but that doens't tell me which goes to which.

Hopefully I wasn't too confusing above. Any help would be appreciated!

Thanks!

TBC Cosmo · 08-26-2009, 03:18 PM

It's been a while since I looked at SLES, but I think there's a /etc/sysconfig/hwconf where you may find the ethernet device PCI IDs.

hybernate20 · 08-26-2009, 04:00 PM

Thanks for the reply! I looked but I'm not seeing that file or anything similar. Could have been in an older version. I'm running 10 SP2.

TBC Cosmo · 08-26-2009, 05:07 PM

Check for a file called /etc/sysconfig/hardware. There is also a command 'hwup' that uses this file. Look at man hwup.

evilted · 08-26-2009, 06:45 PM

you could just plug a cable in one at a time then youd know which interface was up or down..

hybernate20 · 08-26-2009, 07:46 PM

@TBC Cosmo:
Thanks! I'll check that in the morning. I saw that directory, but I never took a look at the hwup script in it.

@evilted:
That still won't tell me which hardware address is assigned to which NIC... that will just tell me which interface is assigned to which NIC. I have to do that regardless to find out which interface is which.

evilted · 08-26-2009, 07:48 PM

actually the output of ifconfig will tell you the mac of the interface...

Code:

 ifconfig

eth1      Link encap:Ethernet  HWaddr 01:02:03:04:05:06

i dont have a command hwup!

Code:

 hwup
-bash: hwup: command not found

I have setup different forms of bonding, and for me all i am interested in is ethx and the port on the back of the machine... since i assign ethx to a mac, and not the name of the network card..

Code:

 ifconfig eth0 hw ether 01:02:03:04:05:06

hybernate20 · 08-27-2009, 07:51 AM

What method are you using for bonding? When I bond, I'm looking for the actual pci hardware address, not the mac address. That is why a simple ifconfig wouldn't do the trick for me.

14:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)

I'm looking at the 00:14:00.1 address. I would be happy to look into other bonding methods if it would simplify the configuration to just needing which ethX it is...

TBC Cosmo · 08-27-2009, 03:17 PM

SLES eth devices are named by the MAC (ifcfg-eth-id-[MAC]), so in the directory, /etc/sysconfig/network:

Code:

sles10-vm1:/etc/sysconfig/network # ls
config  dhcp  if-down.d  if-up.d  ifcfg-eth-id-00:0c:29:14:73:ad  ifcfg-lo  ifcfg.template  ifroute-lo  ifservices.template  providers  scripts

Code:

sles10-vm1:/etc/sysconfig/network # lspci | grep -i eth
02:01.0 Ethernet controller: Advanced Micro Devices [AMD] 79c970 [PCnet32 LANCE] (rev 10)
02:02.0 Ethernet controller: Advanced Micro Devices [AMD] 79c970 [PCnet32 LANCE] (rev 10)
sles10-vm1:/etc/sysconfig/network # grep pci ifcfg-eth-id-00\:0c\:29\:14\:73\:ad
_nm_name='bus-pci-0000:02:01.0'
sles10-vm1:/etc/sysconfig/network # ip address show
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
2: eth0: <BROADCAST,MULTICAST,NOTRAILERS,UP> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:0c:29:14:73:ad brd ff:ff:ff:ff:ff:ff
    inet x.x.x.x/23 brd x.x.x.x scope global eth0
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
    link/ether 00:0c:29:14:73:b7 brd ff:ff:ff:ff:ff:ff

hybernate20 · 08-28-2009, 07:58 AM

I've seen that before, but that requires that you apply some kind of network settings to each interface in yast so that it will create the ifcfg-eth-id-mac files. Otherwise none of those files exist. I wonder where yast gets its information about which mac goes to which hw address and which of those goes to which ethX.

TBC Cosmo · 08-28-2009, 08:07 AM

So let yast create those device files with a default dhcp config, then edit them by hand for bonding?

I know that doesn't address your concern, but it seems more efficient use of time rather than digging into yast underpinnings.

evilted · 08-28-2009, 08:33 AM

Quote:

Originally Posted by hybernate20

What method are you using for bonding? When I bond, I'm looking for the actual pci hardware address, not the mac address. That is why a simple ifconfig wouldn't do the trick for me.

14:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)

I'm looking at the 00:14:00.1 address. I would be happy to look into other bonding methods if it would simplify the configuration to just needing which ethX it is...

take a look here
http://www.cyberciti.biz/tips/linux-...interface.html

its pretty simple, just add the bondx interface to modprobe.conf
alias bond0 bonding
options bond0 mode=balance-alb miimon=100

then in your ifcfg-ethx scripts put a line like

MASTER=bond0

lastly you need to create your bondx scripts which contain real ip's..

the options in modprobe tell what type of bond ur using in the above example its active-load balancing..

but it all depends on the type of bond you require

this definately works on centos. not sure about gentoo/Suse? or other distro's. but ive been running like this on one machine for more than a year, no prob's

also we used to run fedore core. but i migrated a while back to cent, and i will say im not looking back. for a server install its stable and fast. it may not have all the bells and whistles, but for what were doing its the best distro ive used (and i have tried a few). i dont need fancy graphics and pretty things. i want networking, and i want it too be fast and stable, and thats basically it. were running 6 dell servers and 1 HP, and i took the hp offline a few months ago because the damn fans inside it are too loud!

hybernate20 · 08-31-2009, 09:49 AM

That is pretty much how I do it, except that I've been using the

BONDING_SLAVE1=bus-pci-0000:03:00.0
BONDING_SLAVE2=bus-pci-0000:05:00.0

options in the ifcfg-bondX scripts. This way it doesn't require that you make separate ifcfg-ethX scripts for the enslaved interfaces, it just does it via the hardware address. Maybe it would just be easier for me to do it the other way...