LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices


Reply
  Search this Thread
Old 06-12-2014, 12:55 PM   #1
grob115
Member
 
Registered: Oct 2005
Posts: 541

Rep: Reputation: 32
ifconfig reports packet drop


Hi, I'm seeing packet drop on a machine for the receiving end. Looking at a guide here, I have increased the RX ring buffer size. However, even then it's still reporting packet drops.

Two questions:
1) What do I do to troubleshoot or solve this?
2) The sender on the other hand isn't reporting any packet drop. Shouldn't the drop reported by the recipient reported by the sender as well as sent packet drop?

Code:
[~] # ifconfig
eth0      Link encap:Ethernet  HWaddr 00:08:9B:CD:FD:F5
          inet addr:192.168.60.5  Bcast:192.168.60.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3881463 errors:0 dropped:73912 overruns:0 frame:0
          TX packets:4905859 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1564880262 (1.4 GiB)  TX bytes:587640457 (560.4 MiB)
          Interrupt:18 Memory:feae0000-feb00000

eth1      Link encap:Ethernet  HWaddr 00:08:9B:CD:FD:F6
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Interrupt:17 Memory:fe9e0000-fea00000

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:519175 errors:0 dropped:0 overruns:0 frame:0
          TX packets:519175 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:491471327 (468.7 MiB)  TX bytes:491471327 (468.7 MiB)

[~] #
[~] # ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             4096
Current hardware settings:
RX:             256
RX Mini:        0
RX Jumbo:       0
TX:             256

[~] #
[~] # ethtool -G eth0 rx 4096
[~] #
[~] # ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             4096
Current hardware settings:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             256
 
Old 06-12-2014, 03:35 PM   #2
business_kid
LQ Guru
 
Registered: Jan 2006
Location: Ireland
Distribution: Slackware, Pi OS & Android
Posts: 12,018

Rep: Reputation: 1416Reputation: 1416Reputation: 1416Reputation: 1416Reputation: 1416Reputation: 1416Reputation: 1416Reputation: 1416Reputation: 1416Reputation: 1416
Routers will drop packets big time if they have no route for them. I'm seeing no errors, no overruns. So it looks like valid packets are going nowhere. Are you being attacked? Is there a firewall busy? Remember things like nmap scanning your open ports will send one packet per port number.Has anyone reported data loss?

Once you get answers to those questions (usually in logs, but you might find your logging inadequate for your curiosity) you will be able to tell us.
 
Old 06-13-2014, 01:10 PM   #3
stateless
Member
 
Registered: Jan 2013
Distribution: Debian
Posts: 166
Blog Entries: 1

Rep: Reputation: 4
Two thoughts:

* Aren't ifconfig "dropped" packets referring to dropped frames on the link level? (Routing is higher up -- you can have, for example, loss of ping packets without increasing the drop count.) So, I would be inclined to look at the congestion issue, and then local ethernet cabling, cards, and drivers. I noticed this interesting article: http://datacenteroverlords.com/2013/...t-or-pause-it/

* I'd personally avoid enlarging network buffers... this attempt to mask the problem can cause more problems by confusing tcp flow control.
 
Old 06-16-2014, 10:02 AM   #4
grob115
Member
 
Registered: Oct 2005
Posts: 541

Original Poster
Rep: Reputation: 32
Hi, thanks for the answer. Here are some additional details ... The connection is PC -> Firewall -> NAS and the ifconfig was captured on the NAS while a transfer was happening from the PC to the NAS. Both links are 1Gbps connections and the whole thing is on a private network so no attacks here and no data loss has occurred. If the file is being transferred, just a lot slower than expected, does the routing issue still apply?

I checked the /var/log/messages log on the NAS but it's empty.


Quote:
Originally Posted by business_kid View Post
Routers will drop packets big time if they have no route for them. I'm seeing no errors, no overruns. So it looks like valid packets are going nowhere. Are you being attacked? Is there a firewall busy? Remember things like nmap scanning your open ports will send one packet per port number.Has anyone reported data loss?

Once you get answers to those questions (usually in logs, but you might find your logging inadequate for your curiosity) you will be able to tell us.
 
Old 06-16-2014, 10:20 AM   #5
business_kid
LQ Guru
 
Registered: Jan 2006
Location: Ireland
Distribution: Slackware, Pi OS & Android
Posts: 12,018

Rep: Reputation: 1416Reputation: 1416Reputation: 1416Reputation: 1416Reputation: 1416Reputation: 1416Reputation: 1416Reputation: 1416Reputation: 1416Reputation: 1416
Stateless is probably right about dropped packets being at link level --> congestion or other luink issues.

Divide and conquer.
There are a number of situations giving rise to lag. Each rule in the firewall, load on pcs, some network seeing a comms error and dropping back to 10 or 100MB. As that's a 3 machine link, you should be able to check who is seeing lost packets. Have you tried a transfer the other way on that link?
 
Old 06-16-2014, 03:45 PM   #6
nini09
Senior Member
 
Registered: Apr 2009
Posts: 1,737

Rep: Reputation: 154Reputation: 154
NAS access could generate a burst traffic to make CPU one core very busy even if it is 1G speed.
 
Old 06-20-2014, 11:45 AM   #7
grob115
Member
 
Registered: Oct 2005
Posts: 541

Original Poster
Rep: Reputation: 32
Quote:
Originally Posted by business_kid View Post
Stateless is probably right about dropped packets being at link level --> congestion or other luink issues.

Divide and conquer.
There are a number of situations giving rise to lag. Each rule in the firewall, load on pcs, some network seeing a comms error and dropping back to 10 or 100MB. As that's a 3 machine link, you should be able to check who is seeing lost packets. Have you tried a transfer the other way on that link?
Okay thanks. Stateless recommended to look at congestion issue first before anything else. If it's explained in more detail with the link he provided sorry it's not apparent what it is I have to do. So how do I check for congestion issues?

Assuming there are dropped packets on the NAS, the PC should be doing re-transmits? If yes, how can check on the PC end to verify?
 
Old 06-21-2014, 04:08 AM   #8
business_kid
LQ Guru
 
Registered: Jan 2006
Location: Ireland
Distribution: Slackware, Pi OS & Android
Posts: 12,018

Rep: Reputation: 1416Reputation: 1416Reputation: 1416Reputation: 1416Reputation: 1416Reputation: 1416Reputation: 1416Reputation: 1416Reputation: 1416Reputation: 1416
To check for retransmits, send data and check the integrity with diff, sha1sum or md5sum. I would start with ifconfig, note dropped packets, make the transfer, run ifconfig again, and check data integrity.

Congestion you will find with tcpdump, or wireshark. Both are packet sniffers (wireshark needs wine iirc). Both are messy things to use, but they form large logs and you can check these for errors.

You are doing stuff that requires a good deal of network knowledge, and reading is in order if you are going to solve anything. Google is your friend.
 
Old 06-21-2014, 04:30 AM   #9
grob115
Member
 
Registered: Oct 2005
Posts: 541

Original Poster
Rep: Reputation: 32
Quote:
Originally Posted by business_kid View Post
To check for retransmits, send data and check the integrity with diff, sha1sum or md5sum. I would start with ifconfig, note dropped packets, make the transfer, run ifconfig again, and check data integrity.
Sorry I thought checksums will only check if the 2 files are identical. Even if there were re-transmits, the transferred file will ultimately be the same eventually I thought. Is this not the case?

I did run ifconfig and note the packet, and while doing the transfer, continuously doing ifconfig and saw the count increasing.

Quote:
Originally Posted by business_kid View Post
Congestion you will find with tcpdump, or wireshark. Both are packet sniffers (wireshark needs wine iirc). Both are messy things to use, but they form large logs and you can check these for errors.
Just run wireshark but didn't find any re-transmission. In fact, wireshark reported no packet drop!! How can this be?

You can see from the attached screen shots the following:
Windows Explorer reported 10.9MByte/s
WireShark reported 71.1Mbit/s
WireShark reported no packet drop
Attached Thumbnails
Click image for larger version

Name:	WireSharkProtocolHierarchySMB-71.106MBitPerSec.png
Views:	630
Size:	42.6 KB
ID:	15764   Click image for larger version

Name:	WireSharkPacketListNoDrop.png
Views:	561
Size:	150.8 KB
ID:	15765   Click image for larger version

Name:	Windows-10.9MBPerSec.png
Views:	401
Size:	41.2 KB
ID:	15766  
 
Old 06-21-2014, 10:09 AM   #10
business_kid
LQ Guru
 
Registered: Jan 2006
Location: Ireland
Distribution: Slackware, Pi OS & Android
Posts: 12,018

Rep: Reputation: 1416Reputation: 1416Reputation: 1416Reputation: 1416Reputation: 1416Reputation: 1416Reputation: 1416Reputation: 1416Reputation: 1416Reputation: 1416
Huh, that was actually dumb of me. Wireshark only sees received packets. It would love to see the ones that don't make it, but can't.
 
Old 06-21-2014, 10:09 AM   #11
GunFighT
Member
 
Registered: May 2014
Location: Romania
Distribution: Debian, Red Hat
Posts: 52

Rep: Reputation: Disabled
The dropped pakets, on if config, happends when you have a conectivity problem with the cable. Every time when I had this problem it was a imperfect connection at the RJ-45, or cable fault.
You sould check this 1st. There could be CRC (Cyclic redundancy check) errors as well. Try to find out if there are CRC errors.
Hope it helps.
 
Old 06-23-2014, 03:37 PM   #12
nini09
Senior Member
 
Registered: Apr 2009
Posts: 1,737

Rep: Reputation: 154Reputation: 154
Check following staff.

Beginning with kernel 2.6.37, it has been changed the meaning of dropped packet count. Before, dropped packets was most likely due to an error. Now, the rx_dropped counter shows statistics for dropped frames because of:

Softnet backlog full
Bad / Unintended VLAN tags
Unknown / Unregistered protocols
IPv6 frames when the server is not configured for IPv6

If any frames meet those conditions, they are dropped before the protocol stack and the rx_dropped counter is incremented.

Care should be taken to confirm that frames are not being legitimately dropped. A quick way to test this (WARNING: this test does not work for bonding interfaces) is to start a packet capture:

host:~# tcpdump

And then watching the rx_dropped counter. If it stops incrementing while the tcpdump is running; then it is more than likely showing drops because of the reasons listed earlier. If frames continue to be dropped while running tcpdump, investigation should take place to determine root cause.
 
Old 06-24-2014, 11:35 AM   #13
grob115
Member
 
Registered: Oct 2005
Posts: 541

Original Poster
Rep: Reputation: 32
Quote:
Originally Posted by nini09 View Post
Check following staff.

Beginning with kernel 2.6.37, it has been changed the meaning of dropped packet count. Before, dropped packets was most likely due to an error. Now, the rx_dropped counter shows statistics for dropped frames because of:

Softnet backlog full
Bad / Unintended VLAN tags
Unknown / Unregistered protocols
IPv6 frames when the server is not configured for IPv6

If any frames meet those conditions, they are dropped before the protocol stack and the rx_dropped counter is incremented.

Care should be taken to confirm that frames are not being legitimately dropped. A quick way to test this (WARNING: this test does not work for bonding interfaces) is to start a packet capture:

host:~# tcpdump

And then watching the rx_dropped counter. If it stops incrementing while the tcpdump is running; then it is more than likely showing drops because of the reasons listed earlier. If frames continue to be dropped while running tcpdump, investigation should take place to determine root cause.
Sorry can you explain why the drop counter may or may not increment when running tcpdump? I thought tcpdump simply prints to log what's been received, regardless what the system is going to do with the packets.

Also what is Softnet backlog full?
 
Old 06-24-2014, 03:25 PM   #14
nini09
Senior Member
 
Registered: Apr 2009
Posts: 1,737

Rep: Reputation: 154Reputation: 154
The tcpdump change interface setting, such as promiscuous mode. In promiscuous mode, some checks are disabled.
 
Old 07-13-2014, 03:51 AM   #15
grob115
Member
 
Registered: Oct 2005
Posts: 541

Original Poster
Rep: Reputation: 32
Hi, okay got it. I have done the test via ifconfig. And as you have suggested no additional rx packet drop has occurred. So looks like it is due to one of the following reasons? So does this mean it's a network error or what? How can I determine what is causing this? And where can I read more about this change in behavior? I'd imagine a lot of network monitoring tools will suddenly report packet drop out of no where after a Linux OS upgrade. Isn't it?
Softnet backlog full
Bad / Unintended VLAN tags
Unknown / Unregistered protocols
IPv6 frames when the server is not configured for IPv6


1) Started iperf in server mode on the NAS.
Code:
[~] # /opt/bin/iperf -s -m
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 192.168.60.5 port 5001 connected with 192.168.60.2 port 52914
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.3 sec  42.9 MBytes  35.1 Mbits/sec
[  4] MSS size 1160 bytes (MTU 1200 bytes, unknown interface)
2) Started tcpdump and write output to a file.
Code:
[~] # /opt/sbin/tcpdump src 192.168.60.2 -w tcpdump.log
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
^C39311 packets captured
39311 packets received by filter
0 packets dropped by kernel
3) Take note of the count of dropped rx packets on the NAS so far.
Code:
RX packets:1066620 errors:0 dropped:71253 overruns:0 frame:0
4) Started iperf in client mode on my PC.
Code:
iperf.exe -c 192.168.60.5
------------------------------------------------------------
Client connecting to 192.168.60.5, TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.60.2 port 52914 connected with 192.168.60.5 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.2 sec  42.9 MBytes  35.1 Mbits/sec
5) Stopped tcpdump and attempted to read output.
Code:
[~] # /opt/sbin/tcpdump src 192.168.60.2 -r tcpdump.log
reading from file tcpdump.log, link-type EN10MB (Ethernet)
00:22:26.140523 [|ether]
tcpdump: pcap_loop: truncated dump file; tried to read 1214 captured bytes, only got 900
6) Take note of the count of dropped rx packets on the NAS so far.
Code:
RX packets:1106075 errors:0 dropped:71253 overruns:0 frame:0
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
packet drop notifications vlyamtse Linux - Networking 0 12-07-2011 11:32 AM
Packet drop measured by ethtool, tcpdump and ifconfig hjazz6 Linux - Networking 1 05-20-2010 10:19 PM
ifconfig reports millions of dropped RX packets b00n Linux - Networking 7 07-12-2009 02:06 PM
how to delay and drop the packet sonika_singhi Linux - Networking 4 04-02-2005 05:55 PM
drop packet in router becky_starr Programming 0 02-17-2004 09:43 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Networking

All times are GMT -5. The time now is 09:07 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration