LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software > Linux - Kernel
User Name
Password
Linux - Kernel This forum is for all discussion relating to the Linux kernel.

Notices


Reply
  Search this Thread
Old 01-06-2010, 01:31 PM   #1
zman2245
LQ Newbie
 
Registered: Mar 2009
Location: San Francisco, CA
Posts: 26

Rep: Reputation: 15
RedHat: Created extra netdev in ixgbe driver; but cannot ping on its interface


Hey all, sorry for the longish post but I think I need to give sufficient backgroud

I am doing some work which requires 2 interfaces per 10GE port. So I have hacked the ixgbe driver to created an extra netdev object each time probe is invoked. When I insert ixgbe.ko, eth0 & eth1 can be configured on port 0.

Connected to my box is a traffic generator. I can ping it fine using eth0, but not using eth1. In fact, no traffic at all reaches the traffic generator when I try to ping through eth1. I can ping the IP address configured on eth1 itself successfully.

I know that it will be difficult for anyone to give me a perfect diagnosis since I hacked the original driver, but I am looking for pointers on how to debug this. I *think* I am calling all the proper routines when I create and bringup the extra netdev (i.e. netif_carrier_on, netif_tx_start_all_queues, netif_tx_wake_all_queues).

I have included output of ifconfig, iptables -L, and logs from my pings below. The ping using eth1 takes a few seconds and then prints "Destination Host Unreachable."

Thank you so much for any help and advice!
-Zack

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[root@Ares net]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:1B:21:36:09:B4
inet addr:3.0.0.1 Bcast:3.0.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:4 errors:0 dropped:0 overruns:0 frame:0
TX packets:29 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:316 (316.0 b) TX bytes:3588103300 (3.3 GiB)

eth1 Link encap:Ethernet HWaddr 00:1B:21:36:09:B4
inet addr:4.0.0.1 Bcast:4.0.0.255 Mask:255.255.255.0
UP BROADCAST PROMISC MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:143529680 (136.8 MiB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:107 errors:0 dropped:0 overruns:0 frame:0
TX packets:107 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:7800 (7.6 KiB) TX bytes:7800 (7.6 KiB)

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[root@Ares net]# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
[root@Ares net]#

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[root@Ares net]# ping 3.0.0.2
PING 3.0.0.2 (3.0.0.2) 56(84) bytes of data.
64 bytes from 3.0.0.2: icmp_seq=1 ttl=64 time=3.10 ms
64 bytes from 3.0.0.2: icmp_seq=2 ttl=64 time=1.02 ms

--- 3.0.0.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 1.021/2.062/3.104/1.042 ms
[root@Ares net]#
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[root@Ares net]# ping 4.0.0.1
PING 4.0.0.1 (4.0.0.1) 56(84) bytes of data.
64 bytes from 4.0.0.1: icmp_seq=1 ttl=64 time=0.024 ms
64 bytes from 4.0.0.1: icmp_seq=2 ttl=64 time=0.018 ms

--- 4.0.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.018/0.021/0.024/0.003 ms
[root@Ares net]#
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[root@Ares net]# ping 4.0.0.2
PING 4.0.0.2 (4.0.0.2) 56(84) bytes of data.
From 4.0.0.1 icmp_seq=2 Destination Host Unreachable
From 4.0.0.1 icmp_seq=3 Destination Host Unreachable
From 4.0.0.1 icmp_seq=4 Destination Host Unreachable

--- 4.0.0.2 ping statistics ---
5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 4001ms
, pipe 3
[root@Ares net]#
 
Old 01-08-2010, 01:31 PM   #2
Mara
Moderator
 
Registered: Feb 2002
Location: Grenoble
Distribution: Debian
Posts: 9,696

Rep: Reputation: 232Reputation: 232Reputation: 232
Check the following things:
1. If route entry is present for 4.0.0.x
2. Review the netdev structure. It seems to me that the packet reaches the output routines, but it doesn't come back. So it's either a problem at output or the function pointers are incorrect. Especially, check what happens when the device receives rx interrupt or polls for new packet. It may be that you always call the routines for the first netdev .
I would add printk() to all basic routines in the driver to see for which interfaces they are called (if ever).
 
1 members found this post helpful.
Old 01-11-2010, 12:57 PM   #3
zman2245
LQ Newbie
 
Registered: Mar 2009
Location: San Francisco, CA
Posts: 26

Original Poster
Rep: Reputation: 15
Mara,

Thanks for the tips - ended up being a problem with my netif_carrier_on/off calls. The functionality is now fine but I am seeing an issue when rmmod/insmod'ing the driver multtiple times.

Several CPU cores (I have 16 on the board) end up in deadlock. From the stack traces, it looks like several CPUs are trying to write to the journal. The issue occurs after executing "insmod/rmmod ixgbe" a couple times. Why would registering/un-registering netdevices have anything to do with these traces? What could be the connection between the netdev structure and the tracebacks?

I have included the full trace for the first core to report the error and then the first few calls of the traces for a couple other cores....


[root@Ares ixgbe-new]# rmmod ixgbe

BUG: soft lockup - CPU#10 stuck for 10s! [syslogd:3305]
CPU 10:
Modules linked in: ixgbe(U) autofs4(U) dm_multipath(U) scsi_dh(U) video(U) backlight(U) button(U) pata_acpi(U) sg(U) igb(U) ata_generic(U) usbhid(U) i2c_i801(U) pcspkr(U) dm_snapshot(U) dm_zero(U) dm_mirror(U) dm_log(U) dm_mod(U) usb_storage(U) ata_piix(U) libata(U) mptsas(U) mptscsih(U) mptbase(U) scsi_transport_sas(U) sd_mod(U) scsi_mod(U) ext3(U) jbd(U) uhci_hcd(U) ohci_hcd(U) ehci_hcd(U)
Pid: 3305, comm: syslogd Tainted: G 2.6.18-128.el5.cdstv_riobravo #3
RIP: 0010:[<ffffffff80063cde>] [<ffffffff80063cde>] .text.lock.spinlock+0x5/0x30
RSP: 0018:ffff8100809c59e8 EFLAGS: 00000246
RAX: 0000000000000010 RBX: ffff8100809c59e8 RCX: 0000000000002000
RDX: ffff81008a975400 RSI: ffff8100700fed68 RDI: ffff810084a841f0
RBP: ffff81007224c5f8 R08: ffff8100809c5dd8 R09: 000000000000008a
R10: 0000000000000004 R11: 00002b813f8e323f R12: ffffffff80123ea0
R13: ffff8100809c5978 R14: ffffffff80061f5c R15: ffff8100809c5a08
FS: 00002b81400596e0(0000) GS:ffff81008c2e1f40(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002b275f238a60 CR3: 000000008576c000 CR4: 00000000000006e0

Call Trace:
[<ffffffff8003d714>] remove_wait_queue+0x2f/0x38
[<ffffffff88031ffe>] :jbd:start_this_handle+0x25f/0x364
[<ffffffff800e0a8c>] free_poll_entry+0x21/0x25
[<ffffffff8001b9b4>] poll_freewait+0x2a/0x6b
[<ffffffff880321d5>] :jbd:journal_start+0xd2/0x10a
[<ffffffff88055b33>] :ext3:ext3_journal_start_sb+0x4a/0x4c
[<ffffffff880502ed>] :ext3:ext3_prepare_write+0x43/0x185
[<ffffffff80010da3>] generic_file_buffered_write+0x2d0/0x789
[<ffffffff80028e43>] kfree_skbmem+0x7a/0x7e
[<ffffffff8000f0fa>] current_fs_time+0x3f/0x41
[<ffffffff801ec90e>] kfree_skb+0x2b/0x2d
[<ffffffff800175ed>] __generic_file_aio_write_nolock+0x388/0x3d4
[<ffffffff80031d68>] sock_recvmsg+0xf7/0x110
[<ffffffff800c00e1>] __generic_file_write_nolock+0xaf/0xc9
[<ffffffff8009cf02>] autoremove_wake_function+0x0/0x38
[<ffffffff800c0148>] generic_file_writev+0x4d/0xaf
[<ffffffff800d8c95>] do_readv_writev+0x175/0x2a8
[<ffffffff800190df>] do_sync_write+0x0/0x12e
[<ffffffff800d8e09>] vfs_writev+0x41/0x4c
[<ffffffff800d8e5b>] sys_writev+0x47/0x94
[<ffffffff8005e28d>] tracesys+0xd5/0xe0

CPU 13:
Call Trace:
[<ffffffff88031e4c>] :jbd:start_this_handle+0xad/0x364
[<ffffffff880321d5>] :jbd:journal_start+0xd2/0x10a
[<ffffffff88055b33>] :ext3:ext3_journal_start_sb+0x4a/0x4c
[<ffffffff88050bcc>] :ext3:ext3_dirty_inode+0x31/0x83

.
.
.

CPU 14:
Call Trace:
[<ffffffff88031e4c>] :jbd:start_this_handle+0xad/0x364
[<ffffffff880321d5>] :jbd:journal_start+0xd2/0x10a
[<ffffffff88055b33>] :ext3:ext3_journal_start_sb+0x4a/0x4c
[<ffffffff88050bcc>] :ext3:ext3_dirty_inode+0x31/0x83
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Netdev Watchdog Problem, for a Serial Line IP network interface. sharat_chandra Linux - Networking 0 06-27-2008 12:59 PM
Setup extra interface to use with port 80 and 443 jessemirza Linux - Networking 3 03-01-2008 08:17 AM
LXer: Include diff attachment -- Re: Fedora-netdev FC4: kernel-2.6.14-1.1656_FC4.netdev.7 is now available! LXer Syndicated Linux News 0 01-09-2006 08:01 PM
LXer: Fedora-netdev FC4: kernel-2.6.14-1.1653_FC4.netdev.6 is now available! LXer Syndicated Linux News 0 12-15-2005 01:32 PM
LXer: Fedora-netdev FC4: kernel-2.6.14-1.1644_FC4.netdev.5 is now available! LXer Syndicated Linux News 0 12-12-2005 03:01 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software > Linux - Kernel

All times are GMT -5. The time now is 10:49 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration