LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Networking (https://www.linuxquestions.org/questions/linux-networking-3/)
-   -   ifconfig reports packet drop (https://www.linuxquestions.org/questions/linux-networking-3/ifconfig-reports-packet-drop-4175507861/)

grob115 06-12-2014 11:55 AM

ifconfig reports packet drop
 
Hi, I'm seeing packet drop on a machine for the receiving end. Looking at a guide here, I have increased the RX ring buffer size. However, even then it's still reporting packet drops.

Two questions:
1) What do I do to troubleshoot or solve this?
2) The sender on the other hand isn't reporting any packet drop. Shouldn't the drop reported by the recipient reported by the sender as well as sent packet drop?

Code:

[~] # ifconfig
eth0      Link encap:Ethernet  HWaddr 00:08:9B:CD:FD:F5
          inet addr:192.168.60.5  Bcast:192.168.60.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3881463 errors:0 dropped:73912 overruns:0 frame:0
          TX packets:4905859 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1564880262 (1.4 GiB)  TX bytes:587640457 (560.4 MiB)
          Interrupt:18 Memory:feae0000-feb00000

eth1      Link encap:Ethernet  HWaddr 00:08:9B:CD:FD:F6
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Interrupt:17 Memory:fe9e0000-fea00000

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:519175 errors:0 dropped:0 overruns:0 frame:0
          TX packets:519175 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:491471327 (468.7 MiB)  TX bytes:491471327 (468.7 MiB)

[~] #
[~] # ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX:            4096
RX Mini:        0
RX Jumbo:      0
TX:            4096
Current hardware settings:
RX:            256
RX Mini:        0
RX Jumbo:      0
TX:            256

[~] #
[~] # ethtool -G eth0 rx 4096
[~] #
[~] # ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX:            4096
RX Mini:        0
RX Jumbo:      0
TX:            4096
Current hardware settings:
RX:            4096
RX Mini:        0
RX Jumbo:      0
TX:            256


business_kid 06-12-2014 02:35 PM

Routers will drop packets big time if they have no route for them. I'm seeing no errors, no overruns. So it looks like valid packets are going nowhere. Are you being attacked? Is there a firewall busy? Remember things like nmap scanning your open ports will send one packet per port number.Has anyone reported data loss?

Once you get answers to those questions (usually in logs, but you might find your logging inadequate for your curiosity) you will be able to tell us.

stateless 06-13-2014 12:10 PM

Two thoughts:

* Aren't ifconfig "dropped" packets referring to dropped frames on the link level? (Routing is higher up -- you can have, for example, loss of ping packets without increasing the drop count.) So, I would be inclined to look at the congestion issue, and then local ethernet cabling, cards, and drivers. I noticed this interesting article: http://datacenteroverlords.com/2013/...t-or-pause-it/

* I'd personally avoid enlarging network buffers... this attempt to mask the problem can cause more problems by confusing tcp flow control.

grob115 06-16-2014 09:02 AM

Hi, thanks for the answer. Here are some additional details ... The connection is PC -> Firewall -> NAS and the ifconfig was captured on the NAS while a transfer was happening from the PC to the NAS. Both links are 1Gbps connections and the whole thing is on a private network so no attacks here and no data loss has occurred. If the file is being transferred, just a lot slower than expected, does the routing issue still apply?

I checked the /var/log/messages log on the NAS but it's empty.


Quote:

Originally Posted by business_kid (Post 5187203)
Routers will drop packets big time if they have no route for them. I'm seeing no errors, no overruns. So it looks like valid packets are going nowhere. Are you being attacked? Is there a firewall busy? Remember things like nmap scanning your open ports will send one packet per port number.Has anyone reported data loss?

Once you get answers to those questions (usually in logs, but you might find your logging inadequate for your curiosity) you will be able to tell us.


business_kid 06-16-2014 09:20 AM

Stateless is probably right about dropped packets being at link level --> congestion or other luink issues.

Divide and conquer.
There are a number of situations giving rise to lag. Each rule in the firewall, load on pcs, some network seeing a comms error and dropping back to 10 or 100MB. As that's a 3 machine link, you should be able to check who is seeing lost packets. Have you tried a transfer the other way on that link?

nini09 06-16-2014 02:45 PM

NAS access could generate a burst traffic to make CPU one core very busy even if it is 1G speed.

grob115 06-20-2014 10:45 AM

Quote:

Originally Posted by business_kid (Post 5188848)
Stateless is probably right about dropped packets being at link level --> congestion or other luink issues.

Divide and conquer.
There are a number of situations giving rise to lag. Each rule in the firewall, load on pcs, some network seeing a comms error and dropping back to 10 or 100MB. As that's a 3 machine link, you should be able to check who is seeing lost packets. Have you tried a transfer the other way on that link?

Okay thanks. Stateless recommended to look at congestion issue first before anything else. If it's explained in more detail with the link he provided sorry it's not apparent what it is I have to do. So how do I check for congestion issues?

Assuming there are dropped packets on the NAS, the PC should be doing re-transmits? If yes, how can check on the PC end to verify?

business_kid 06-21-2014 03:08 AM

To check for retransmits, send data and check the integrity with diff, sha1sum or md5sum. I would start with ifconfig, note dropped packets, make the transfer, run ifconfig again, and check data integrity.

Congestion you will find with tcpdump, or wireshark. Both are packet sniffers (wireshark needs wine iirc). Both are messy things to use, but they form large logs and you can check these for errors.

You are doing stuff that requires a good deal of network knowledge, and reading is in order if you are going to solve anything. Google is your friend.

grob115 06-21-2014 03:30 AM

3 Attachment(s)
Quote:

Originally Posted by business_kid (Post 5191606)
To check for retransmits, send data and check the integrity with diff, sha1sum or md5sum. I would start with ifconfig, note dropped packets, make the transfer, run ifconfig again, and check data integrity.

Sorry I thought checksums will only check if the 2 files are identical. Even if there were re-transmits, the transferred file will ultimately be the same eventually I thought. Is this not the case?

I did run ifconfig and note the packet, and while doing the transfer, continuously doing ifconfig and saw the count increasing.

Quote:

Originally Posted by business_kid (Post 5191606)
Congestion you will find with tcpdump, or wireshark. Both are packet sniffers (wireshark needs wine iirc). Both are messy things to use, but they form large logs and you can check these for errors.

Just run wireshark but didn't find any re-transmission. In fact, wireshark reported no packet drop!! How can this be?

You can see from the attached screen shots the following:
Windows Explorer reported 10.9MByte/s
WireShark reported 71.1Mbit/s
WireShark reported no packet drop

business_kid 06-21-2014 09:09 AM

Huh, that was actually dumb of me. Wireshark only sees received packets. It would love to see the ones that don't make it, but can't.

GunFighT 06-21-2014 09:09 AM

The dropped pakets, on if config, happends when you have a conectivity problem with the cable. Every time when I had this problem it was a imperfect connection at the RJ-45, or cable fault.
You sould check this 1st. There could be CRC (Cyclic redundancy check) errors as well. Try to find out if there are CRC errors.
Hope it helps.

nini09 06-23-2014 02:37 PM

Check following staff.

Beginning with kernel 2.6.37, it has been changed the meaning of dropped packet count. Before, dropped packets was most likely due to an error. Now, the rx_dropped counter shows statistics for dropped frames because of:

Softnet backlog full
Bad / Unintended VLAN tags
Unknown / Unregistered protocols
IPv6 frames when the server is not configured for IPv6

If any frames meet those conditions, they are dropped before the protocol stack and the rx_dropped counter is incremented.

Care should be taken to confirm that frames are not being legitimately dropped. A quick way to test this (WARNING: this test does not work for bonding interfaces) is to start a packet capture:

host:~# tcpdump

And then watching the rx_dropped counter. If it stops incrementing while the tcpdump is running; then it is more than likely showing drops because of the reasons listed earlier. If frames continue to be dropped while running tcpdump, investigation should take place to determine root cause.

grob115 06-24-2014 10:35 AM

Quote:

Originally Posted by nini09 (Post 5192737)
Check following staff.

Beginning with kernel 2.6.37, it has been changed the meaning of dropped packet count. Before, dropped packets was most likely due to an error. Now, the rx_dropped counter shows statistics for dropped frames because of:

Softnet backlog full
Bad / Unintended VLAN tags
Unknown / Unregistered protocols
IPv6 frames when the server is not configured for IPv6

If any frames meet those conditions, they are dropped before the protocol stack and the rx_dropped counter is incremented.

Care should be taken to confirm that frames are not being legitimately dropped. A quick way to test this (WARNING: this test does not work for bonding interfaces) is to start a packet capture:

host:~# tcpdump

And then watching the rx_dropped counter. If it stops incrementing while the tcpdump is running; then it is more than likely showing drops because of the reasons listed earlier. If frames continue to be dropped while running tcpdump, investigation should take place to determine root cause.

Sorry can you explain why the drop counter may or may not increment when running tcpdump? I thought tcpdump simply prints to log what's been received, regardless what the system is going to do with the packets.

Also what is Softnet backlog full?

nini09 06-24-2014 02:25 PM

The tcpdump change interface setting, such as promiscuous mode. In promiscuous mode, some checks are disabled.

grob115 07-13-2014 02:51 AM

Hi, okay got it. I have done the test via ifconfig. And as you have suggested no additional rx packet drop has occurred. So looks like it is due to one of the following reasons? So does this mean it's a network error or what? How can I determine what is causing this? And where can I read more about this change in behavior? I'd imagine a lot of network monitoring tools will suddenly report packet drop out of no where after a Linux OS upgrade. Isn't it?
Softnet backlog full
Bad / Unintended VLAN tags
Unknown / Unregistered protocols
IPv6 frames when the server is not configured for IPv6


1) Started iperf in server mode on the NAS.
Code:

[~] # /opt/bin/iperf -s -m
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 192.168.60.5 port 5001 connected with 192.168.60.2 port 52914
[ ID] Interval      Transfer    Bandwidth
[  4]  0.0-10.3 sec  42.9 MBytes  35.1 Mbits/sec
[  4] MSS size 1160 bytes (MTU 1200 bytes, unknown interface)

2) Started tcpdump and write output to a file.
Code:

[~] # /opt/sbin/tcpdump src 192.168.60.2 -w tcpdump.log
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
^C39311 packets captured
39311 packets received by filter
0 packets dropped by kernel

3) Take note of the count of dropped rx packets on the NAS so far.
Code:

RX packets:1066620 errors:0 dropped:71253 overruns:0 frame:0
4) Started iperf in client mode on my PC.
Code:

iperf.exe -c 192.168.60.5
------------------------------------------------------------
Client connecting to 192.168.60.5, TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.60.2 port 52914 connected with 192.168.60.5 port 5001
[ ID] Interval      Transfer    Bandwidth
[  3]  0.0-10.2 sec  42.9 MBytes  35.1 Mbits/sec

5) Stopped tcpdump and attempted to read output.
Code:

[~] # /opt/sbin/tcpdump src 192.168.60.2 -r tcpdump.log
reading from file tcpdump.log, link-type EN10MB (Ethernet)
00:22:26.140523 [|ether]
tcpdump: pcap_loop: truncated dump file; tried to read 1214 captured bytes, only got 900

6) Take note of the count of dropped rx packets on the NAS so far.
Code:

RX packets:1106075 errors:0 dropped:71253 overruns:0 frame:0


All times are GMT -5. The time now is 10:56 PM.