LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices


Reply
 
Search this Thread
Old 07-03-2013, 09:54 AM   #1
jnielsen7
LQ Newbie
 
Registered: Feb 2013
Posts: 23

Rep: Reputation: Disabled
Unlabeled packets being dropped (reported by dropwatch)


After a recent expansion of our GPFS storage servers (eight new NSD servers in another building) in addition to our existing four NSD [Network Storage Device] servers we had been using we are now seeing some network slowdowns which netstat and tcpdump leads us to believe is due to a high number of retransmits.

After googling around a bit I discovered a tool available on RHEL/Centos systems called dropwatch which will tell you where in the kernel something is being dropped. I'm not sure about the relationship between dropped and retransmitted packets but I got some interesting results from dropwatch which are peculiar to only the NSD servers serving GPFS. We are still investigating our switches but we noticed that certain GPFS and sysctl performance tweaks led to better throughput, so it might be host side.

A typical (Centos) system in our network (I tried this on a few different servers) will have some, what I assume is normal, drops like this (within a 5 second period):

Code:
2 drops at unix_stream_connect+1dc (0xffffffff814bdf1c)
1 drops at netlink_attachskb+398 (0xffffffff81454338)
13 drops at unix_dgram_poll+4cd (0xffffffff814bbf9d)
4 drops at unix_stream_connect+1dc (0xffffffff814bdf1c)
1 drops at netlink_unicast+251 (0xffffffff81454981)
1 drops at audit_log_lost+10b (0xffffffff810caf4b)
2 drops at neigh_event_ns+1b4 (0xffffffff81438544)
1 drops at netlink_attachskb+398 (0xffffffff81454338)
4 drops at unix_dgram_poll+4cd (0xffffffff814bbf9d)
5 drops at __netlink_create+e9 (0xffffffff81452bb9)
5 drops at netlink_unicast+251 (0xffffffff81454981)
1 drops at neigh_event_ns+1b4 (0xffffffff81438544)
8 drops at unix_dgram_poll+4cd (0xffffffff814bbf9d)
4 drops at unix_stream_connect+1dc (0xffffffff814bdf1c)
1 drops at arp_error_report+39 (0xffffffff8148c9a9)
1 drops at netlink_attachskb+398 (0xffffffff81454338)
The NSD Servers serving GPFS (four Redhat 5.8 + eight Centos 6.2), however, show something different consistently (with high numbers) which appears on none of the other servers: netlabel_unlabel_acceptflg. I see this pretty much on all the NSD servers (within a 5 second period):

Code:
1 drops at netlink_run_queue+ff
1 drops at netlink_broadcast+283
62 drops at netlabel_unlabel_acceptflg+7e4f032
64 drops at netlabel_unlabel_acceptflg+7e4f032
64 drops at netlabel_unlabel_acceptflg+7e4f032
1 drops at unix_stream_recvmsg+3fa
1 drops at unix_stream_recvmsg+3fa
16 drops at netlabel_unlabel_acceptflg+7e4f032
64 drops at netlabel_unlabel_acceptflg+7e4f032
64 drops at netlabel_unlabel_acceptflg+7e4f032
1 drops at unix_stream_recvmsg+3fa
2 drops at unix_stream_recvmsg+3fa
64 drops at netlabel_unlabel_acceptflg+7e4f032
1 drops at unix_stream_recvmsg+3fa
64 drops at netlabel_unlabel_acceptflg+7e4f032
57 drops at netlabel_unlabel_acceptflg+7e4f032
7 drops at netlabel_unlabel_acceptflg+7e52827
64 drops at netlabel_unlabel_acceptflg+7e4f032
2 drops at unix_stream_recvmsg+3fa
1 drops at unix_stream_recvmsg+3fa
64 drops at netlabel_unlabel_acceptflg+7e4f032
64 drops at netlabel_unlabel_acceptflg+7e4f032
1 drops at unix_stream_recvmsg+3fa
64 drops at netlabel_unlabel_acceptflg+7e4f032
64 drops at netlabel_unlabel_acceptflg+7e4f032
3 drops at netlink_broadcast+283
1 drops at unix_stream_recvmsg+3fa
2 drops at unix_stream_recvmsg+3fa
2 drops at netlink_broadcast+283
3 drops at unix_stream_recvmsg+3fa
64 drops at netlabel_unlabel_acceptflg+7e4f032
64 drops at netlabel_unlabel_acceptflg+7e4f032
11 drops at unix_stream_recvmsg+3fa
10 drops at unix_dgram_connect+531
1 drops at netlink_destroy_callback+11
5 drops at tcp_rcv_state_process+66
1 drops at unix_stream_recvmsg+3fa
I googled netlabel_unlabel_acceptflg and all I came up with was something about "unlabeled packets" so I started searching on that and it led me to an Oracle page discussing RIPSO/CIPSO and packet labeling: http://docs.oracle.com/cd/E19109-01/...dhn/index.html. I have no idea if that is even related but I am really curious as to what the high number of drops at netlabel_unlabel_acceptflg indicates.

Any ideas or guesses?

Last edited by jnielsen7; 07-03-2013 at 10:07 AM.
 
Old 07-05-2013, 11:41 AM   #2
jnielsen7
LQ Newbie
 
Registered: Feb 2013
Posts: 23

Original Poster
Rep: Reputation: Disabled
What is an unlabeled packet? Drivers or NIC involved?

Does anyone even have a clue what "unlabeled packet" means/what an unlabeled packet is? I for one have never heard of that before searching online for it.

The RIPSO/CIPSO stuff that I looked at seemed like it was connected to IPSEC in some way but as far as I know we are not implementing anything IPSEC related on our network (beyond defaults that may be set without one even knowing it). If this were caused by Default VLAN 1 "noise" broadcast from some other part of the network I would expect to see it on all our servers, but I do not. Maybe it's specific to the type of networking card we have in those RHEL servers (a Chelsio NIC) since this is an OS/kernel message (possibly related to how it's kernel module or driver for the card handles packets).

Code:
dmesg | grep -i chels
Chelsio T3 Network Driver - version 1.1.3-ko-1
eth2: Chelsio T310 10GBASE-CX4 RNIC (rev 4) PCI Express x8 MSI-X
Chelsio T3 iSCSI Driver cxgb3i v2.0.0 (Jun. 2010)
iw_cxgb3: Chelsio T3 RDMA Driver - version 1.1

lspci | grep -i chels
15:00.0 Ethernet controller: Chelsio Communications Inc T310 10GbE Single Port Adapter
Any guesses?

Edit: P.S. I found the code that handles netlabel_unlabel_acceptflg here: http://lxr.free-electrons.com/source...labeled.c#L756 (Lines 756-770). At the top of the file it says: "NetLabel Unlabeled Support - This file defines functions for dealing with unlabeled packets for the NetLabel system. The NetLabel system manages static and dynamic label mappings for network protocols such as CIPSO and RIPSO." Hmm...

Last edited by jnielsen7; 07-05-2013 at 12:07 PM.
 
  


Reply

Tags
drops, packets


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Dropped packets reported by ifconfig - implication? rylan76 Linux - Networking 5 12-31-2006 03:48 AM
Dropped packets Doolspin Linux - Software 1 10-22-2006 01:22 PM
too much dropped packets...Hi.. alaios Linux - Networking 2 02-10-2005 04:49 AM
Dropped packets - is this a problem?? benr77 Linux - General 4 10-04-2004 02:05 PM
dropped packets... sohmc Linux - Software 3 05-29-2003 09:26 AM


All times are GMT -5. The time now is 01:41 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration