Linux - NetworkingThis forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
I am trying to setup lustre environment but facing the issue related to
infiniband setup:
While trying to bring up ib0 I am running:
[root at slave3 ~]# /etc/rc.d/init.d/rdma restart
Unloading OpenIB kernel modules:
Found opensm running.
Please stop all RDMA applications before downing the stack.
[FAILED]
Loading OpenIB kernel modules:FATAL: Error inserting ib_addr
(/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/kernel/drivers/infiniband/core/ib_addr.ko):
Unknown symbol in module, or unknown parameter (see dmesg)
Failed to load module WARNING: Error inserting ib_core
(/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/kernel/drivers/infiniband/core/ib_core.ko):
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting ib_mad
(/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/kernel/drivers/infiniband/core/ib_mad.ko):
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting ib_sa
(/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/kernel/drivers/infiniband/core/ib_sa.ko):
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting iw_cm
(/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/kernel/drivers/infiniband/core/iw_cm.ko):
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting ib_cm
(/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/kernel/drivers/infiniband/core/ib_cm.ko):
Unknown symbol in module, or unknown parameter (see dmesg)
FATAL: Error inserting rdma_cm
(/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/kernel/drivers/infiniband/core/rdma_cm.ko):
Unknown symbol in module, or unknown parameter (see dmesg)
Failed to load module WARNING: Error inserting iw_cm
(/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/kernel/drivers/infiniband/core/iw_cm.ko):
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting ib_cm
(/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/kernel/drivers/infiniband/core/ib_cm.ko):
Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting rdma_cm
(/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/kernel/drivers/infiniband/core/rdma_cm.ko):
Unknown symbol in module, or unknown parameter (see dmesg)
FATAL: Error inserting rdma_ucm
(/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/kernel/drivers/infiniband/core/rdma_ucm.ko):
Unknown symbol in module, or unknown parameter (see dmesg)
Failed to load module FATAL: Error inserting ib_ipoib
(/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko):
Unknown symbol in module, or unknown parameter (see dmesg)
Failed to load module [FAILED]
FYI..
Though I checked that ibstatus and ibstat working fine:
[root at slave3 ~]# ibstat
CA 'mlx4_0'
CA type: MT26428
Number of ports: 1
Firmware version: 2.9.1000
Hardware version: b0
Node GUID: 0x0002c903000be516
System image GUID: 0x0002c903000be519
Port 1:
State: Active
Physical state: LinkUp
Rate: 40
Base lid: 7
LMC: 0
SM lid: 2
Capability mask: 0x0251086a
Port GUID: 0x0002c903000be517
Link layer: InfiniBand
[root at slave3 ~]#
[root at slave3 ~]# ibstatus
Infiniband device 'mlx4_0' port 1 status:
default gid: fe80:0000:0000:0000:0002:c903:000b:e517
base lid: 0x7
sm lid: 0x2
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 40 Gb/sec (4X QDR)
link_layer: InfiniBand
While I rebooted the machine and its up the dmesg was:
Code:
alloc irq_2_iommu on node -1
ioatdma 0000:00:04.1: irq 79 for MSI/MSI-X
ioatdma 0000:00:04.2: PCI INT C -> GSI 31 (level, low) -> IRQ 31
ioatdma 0000:00:04.2: setting latency timer to 64
alloc irq_desc for 80 on node -1
alloc kstat_irqs on node -1
alloc irq_2_iommu on node -1
ioatdma 0000:00:04.2: irq 80 for MSI/MSI-X
ioatdma 0000:00:04.3: PCI INT D -> GSI 39 (level, low) -> IRQ 39
ioatdma 0000:00:04.3: setting latency timer to 64
alloc irq_desc for 81 on node -1
alloc kstat_irqs on node -1
alloc irq_2_iommu on node -1
ioatdma 0000:00:04.3: irq 81 for MSI/MSI-X
ioatdma 0000:00:04.4: PCI INT A -> GSI 31 (level, low) -> IRQ 31
ioatdma 0000:00:04.4: setting latency timer to 64
alloc irq_desc for 82 on node -1
alloc kstat_irqs on node -1
alloc irq_2_iommu on node -1
ioatdma 0000:00:04.4: irq 82 for MSI/MSI-X
ioatdma 0000:00:04.5: PCI INT B -> GSI 39 (level, low) -> IRQ 39
ioatdma 0000:00:04.5: setting latency timer to 64
alloc irq_desc for 83 on node -1
alloc kstat_irqs on node -1
alloc irq_2_iommu on node -1
ioatdma 0000:00:04.5: irq 83 for MSI/MSI-X
ioatdma 0000:00:04.6: PCI INT C -> GSI 31 (level, low) -> IRQ 31
ioatdma 0000:00:04.6: setting latency timer to 64
alloc irq_desc for 84 on node -1
alloc kstat_irqs on node -1
alloc irq_2_iommu on node -1
ioatdma 0000:00:04.6: irq 84 for MSI/MSI-X
ioatdma 0000:00:04.7: PCI INT D -> GSI 39 (level, low) -> IRQ 39
ioatdma 0000:00:04.7: setting latency timer to 64
alloc irq_desc for 85 on node -1
alloc kstat_irqs on node -1
alloc irq_2_iommu on node -1
ioatdma 0000:00:04.7: irq 85 for MSI/MSI-X
mlx4_core: Mellanox ConnectX core driver v1.1 (Dec, 2011)
mlx4_core: Initializing 0000:05:00.0
mlx4_core 0000:05:00.0: PCI INT A -> GSI 32 (level, low) -> IRQ 32
mlx4_core 0000:05:00.0: setting latency timer to 64
udev: renamed network interface eth0 to em1
alloc irq_desc for 86 on node -1
alloc kstat_irqs on node -1
alloc irq_2_iommu on node -1
mlx4_core 0000:05:00.0: irq 86 for MSI/MSI-X
alloc irq_desc for 87 on node -1
alloc kstat_irqs on node -1
alloc irq_2_iommu on node -1
mlx4_core 0000:05:00.0: irq 87 for MSI/MSI-X
alloc irq_desc for 88 on node -1
alloc kstat_irqs on node -1
alloc irq_2_iommu on node -1
mlx4_core 0000:05:00.0: irq 88 for MSI/MSI-X
alloc irq_desc for 89 on node -1
alloc kstat_irqs on node -1
alloc irq_2_iommu on node -1
mlx4_core 0000:05:00.0: irq 89 for MSI/MSI-X
alloc irq_desc for 90 on node -1
alloc kstat_irqs on node -1
alloc irq_2_iommu on node -1
mlx4_core 0000:05:00.0: irq 90 for MSI/MSI-X
alloc irq_desc for 91 on node -1
alloc kstat_irqs on node -1
alloc irq_2_iommu on node -1
mlx4_core 0000:05:00.0: irq 91 for MSI/MSI-X
alloc irq_desc for 92 on node -1
alloc kstat_irqs on node -1
alloc irq_2_iommu on node -1
mlx4_core 0000:05:00.0: irq 92 for MSI/MSI-X
alloc irq_desc for 93 on node -1
alloc kstat_irqs on node -1
alloc irq_2_iommu on node -1
mlx4_core 0000:05:00.0: irq 93 for MSI/MSI-X
alloc irq_desc for 94 on node -1
alloc kstat_irqs on node -1
alloc irq_2_iommu on node -1
mlx4_core 0000:05:00.0: irq 94 for MSI/MSI-X
alloc irq_desc for 95 on node -1
alloc kstat_irqs on node -1
alloc irq_2_iommu on node -1
mlx4_core 0000:05:00.0: irq 95 for MSI/MSI-X
alloc irq_desc for 96 on node -1
alloc kstat_irqs on node -1
alloc irq_2_iommu on node -1
mlx4_core 0000:05:00.0: irq 96 for MSI/MSI-X
alloc irq_desc for 97 on node -1
alloc kstat_irqs on node -1
alloc irq_2_iommu on node -1
mlx4_core 0000:05:00.0: irq 97 for MSI/MSI-X
alloc irq_desc for 98 on node -1
alloc kstat_irqs on node -1
alloc irq_2_iommu on node -1
mlx4_core 0000:05:00.0: irq 98 for MSI/MSI-X
shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
iTCO_vendor_support: vendor-support=0
iTCO_wdt: Intel TCO WatchDog Timer Driver v1.07rh
iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS
i801_smbus 0000:00:1f.3: PCI INT C -> GSI 19 (level, low) -> IRQ 19
ACPI: resource 0000:00:1f.3 [io 0x4000-0x401f] conflicts with ACPI region SMBI [io 0x4000-0x400f]
ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
EDAC MC: Ver: 2.1.0 Dec 14 2012
EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
EDAC MC0: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#0': DEV 0000:ff:0e.0
EDAC sbridge: Driver loaded.
sr0: scsi3-mmc drive: 24x/24x cd/rw xa/form2 cdda tray
Uniform CD-ROM driver Revision: 3.20
sr 6:0:0:0: Attached scsi CD-ROM sr0
microcode: CPU0 sig=0x206d5, pf=0x1, revision=0x513
platform microcode: firmware: requesting intel-ucode/06-2d-05
microcode: CPU1 sig=0x206d5, pf=0x1, revision=0x513
platform microcode: firmware: requesting intel-ucode/06-2d-05
microcode: CPU2 sig=0x206d5, pf=0x1, revision=0x513
platform microcode: firmware: requesting intel-ucode/06-2d-05
microcode: CPU3 sig=0x206d5, pf=0x1, revision=0x513
platform microcode: firmware: requesting intel-ucode/06-2d-05
microcode: CPU4 sig=0x206d5, pf=0x1, revision=0x513
platform microcode: firmware: requesting intel-ucode/06-2d-05
microcode: CPU5 sig=0x206d5, pf=0x1, revision=0x513
platform microcode: firmware: requesting intel-ucode/06-2d-05
microcode: CPU6 sig=0x206d5, pf=0x1, revision=0x513
platform microcode: firmware: requesting intel-ucode/06-2d-05
microcode: CPU7 sig=0x206d5, pf=0x1, revision=0x513
platform microcode: firmware: requesting intel-ucode/06-2d-05
Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
dcdbas dcdbas: Dell Systems Management Base Driver (version 5.6.0-3.2)
sd 1:0:0:0: Attached scsi generic sg0 type 0
sr 6:0:0:0: Attached scsi generic sg1 type 5
EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts:
EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts:
Adding 4030456k swap on /dev/mapper/VolGroup-lv_swap. Priority:-1 extents:1 across:4030456k
igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
802.1Q VLAN Support v1.8 Ben Greear <greearb@candelatech.com>
All bugs added by David S. Miller <davem@redhat.com>
8021q: adding VLAN 0 to HW filter on device eth1
cnic: Unknown symbol ip6_route_output
8021q: adding VLAN 0 to HW filter on device em1
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
type=1305 audit(1368108959.718:48954): auid=4294967295 ses=4294967295 op="remove rule" key=(null) list=4 res=1
type=1305 audit(1368108959.718:48955): audit_enabled=0 old=1 auid=4294967295 ses=4294967295 res=1
readahead-collector: sorting
readahead-collector: finished
While I run /etc/rc.d/init.d/rdma restart
Code:
[root@slave3 ~]# /etc/rc.d/init.d/rdma restart
Unloading OpenIB kernel modules: [ OK ]
Loading OpenIB kernel modules:FATAL: Error inserting ib_addr (/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/kernel/drivers/infiniband/core/ib_addr.ko): Unknown symbol in module, or unknown parameter (see dmesg)
Failed to load module WARNING: Error inserting ib_core (/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/kernel/drivers/infiniband/core/ib_core.ko): Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting ib_mad (/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/kernel/drivers/infiniband/core/ib_mad.ko): Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting ib_sa (/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/kernel/drivers/infiniband/core/ib_sa.ko): Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting iw_cm (/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/kernel/drivers/infiniband/core/iw_cm.ko): Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting ib_cm (/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/kernel/drivers/infiniband/core/ib_cm.ko): Unknown symbol in module, or unknown parameter (see dmesg)
FATAL: Error inserting rdma_cm (/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/kernel/drivers/infiniband/core/rdma_cm.ko): Unknown symbol in module, or unknown parameter (see dmesg)
Failed to load module WARNING: Error inserting iw_cm (/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/kernel/drivers/infiniband/core/iw_cm.ko): Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting ib_cm (/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/kernel/drivers/infiniband/core/ib_cm.ko): Unknown symbol in module, or unknown parameter (see dmesg)
WARNING: Error inserting rdma_cm (/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/kernel/drivers/infiniband/core/rdma_cm.ko): Unknown symbol in module, or unknown parameter (see dmesg)
FATAL: Error inserting rdma_ucm (/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/kernel/drivers/infiniband/core/rdma_ucm.ko): Unknown symbol in module, or unknown parameter (see dmesg)
Failed to load module FATAL: Error inserting ib_ipoib (/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko): Unknown symbol in module, or unknown parameter (see dmesg)
Failed to load module [FAILED]
[root@slave3 ~]#
Code:
dmesg shows:
dcdbas dcdbas: Dell Systems Management Base Driver (version 5.6.0-3.2)
sd 1:0:0:0: Attached scsi generic sg0 type 0
sr 6:0:0:0: Attached scsi generic sg1 type 5
EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts:
EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts:
Adding 4030456k swap on /dev/mapper/VolGroup-lv_swap. Priority:-1 extents:1 across:4030456k
igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
802.1Q VLAN Support v1.8 Ben Greear <greearb@candelatech.com>
All bugs added by David S. Miller <davem@redhat.com>
8021q: adding VLAN 0 to HW filter on device eth1
cnic: Unknown symbol ip6_route_output
8021q: adding VLAN 0 to HW filter on device em1
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
type=1305 audit(1368108959.718:48954): auid=4294967295 ses=4294967295 op="remove rule" key=(null) list=4 res=1
type=1305 audit(1368108959.718:48955): audit_enabled=0 old=1 auid=4294967295 ses=4294967295 res=1
readahead-collector: sorting
readahead-collector: finished
mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (April 4, 2008)
ib_addr: Unknown symbol ipv6_dev_get_saddr
ib_addr: Unknown symbol ip6_route_output
ib_addr: Unknown symbol ipv6_chk_addr
ib_addr: Unknown symbol ipv6_dev_get_saddr
ib_addr: Unknown symbol ip6_route_output
ib_addr: Unknown symbol ipv6_chk_addr
ib_addr: Unknown symbol ipv6_dev_get_saddr
ib_addr: Unknown symbol ip6_route_output
ib_addr: Unknown symbol ipv6_chk_addr
ib_ipoib: Unknown symbol icmpv6_send
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
