Ethernet RX overruns and dropped packets

gosssamer · 05-31-2018, 02:09 PM

Can someone explain why an interface would start showing dropped
packets? I have a fedora28 system with one ethernet interface directly connected to a 65mbit cable connection and the other on a gigabit LAN.

There is another f25 box directly connected to the same modem and it's also experiencing the same dropped packet problem.

The system is generally idle (I don't think it's dropping packets because they can't be processed fast enough), and the cable company (Optimum) have replaced the router/modem and even the line to the pole (we were recently having actual connection problems.)

I've also booted a sysrescue CDROM on my laptop and it also happens with just regular traffic activity.

I believe this has been going on for quite some time.

Could it be some kernel tuning parameter that's affecting both boxes? I don't think it's bad hardware or a slow processor or anything related to a specific component, as it's occurring on two separate machines.

I've searched tons of other posts on similar subjects, but there's rarely a solution. Should I post on the kernel list?

business_kid · 06-02-2018, 02:49 AM

One thing you haven't mentioned is the internal connection. Too long a length of cat5 or cat6 affects the max speed, as does other more exotic factors.

gosssamer · 06-03-2018, 08:26 AM

Quote:

Originally Posted by business_kid

One thing you haven't mentioned is the internal connection. Too long a length of cat5 or cat6 affects the max speed, as does other more exotic factors.

This is the connection coming from the cable modem. There are two 3' CAT5 (maybe even CAT6) cables to each of the two machines. The modem is right next to the systems.

I've also tried changing cables. It happens on all three machines that are connected.

AwesomeMachine · 06-04-2018, 01:06 AM

How are you identifying the packet loss? Do you have 2 NICs on your Fedora28 machine? I've never heard of a 65mbit connection.

gosssamer · 06-05-2018, 09:44 AM

Quote:

Originally Posted by AwesomeMachine

How are you identifying the packet loss? Do you have 2 NICs on your Fedora28 machine? I've never heard of a 65mbit connection.

I'm sorry I was unclear; this is according to the ifconfig output:

Code:

# ifconfig br0
br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 68.195.193.42  netmask 255.255.255.248  broadcast 68.195.193.47
        inet6 fe80::6c26:fdff:fe03:7f7e  prefixlen 64  scopeid 0x20<link>
        inet6 ::ec4:7aff:fea9:18de  prefixlen 64  scopeid 0x0<global>
        ether 0c:c4:7a:a9:18:de  txqueuelen 500  (Ethernet)
        RX packets 134269659  bytes 166191398263 (154.7 GiB)
        RX errors 0  dropped 259885  overruns 0  frame 0
        TX packets 87896153  bytes 66717606784 (62.1 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

This is a gigabit ethernet connection to a cable modem with a 65mbit uplink.

It drops packets constantly, but pings never fail. They are always in the 10ms range to sites like google (172.217.10.110).

So in six days it's dropped about 260k packets:

Code:

# uptime
 10:44:21 up 6 days, 32 min,  2 users,  load average: 0.42, 0.52, 0.56

business_kid · 06-06-2018, 02:14 AM

Exactly where are the packets being lost? Have you tried unplugging the modem, testing as far as the modem (e.g. with another box there temporarily)?

The ISPs will usually replace the router without hassle rather than thinking about your problem. Once the outgoing pipework looks ok, then it's the ISP's problem. Where exactly are you losing the packets?

gosssamer · 06-06-2018, 11:43 AM

Quote:

Originally Posted by business_kid

Exactly where are the packets being lost? Have you tried unplugging the modem, testing as far as the modem (e.g. with another box there temporarily)?

The ISPs will usually replace the router without hassle rather than thinking about your problem. Once the outgoing pipework looks ok, then it's the ISP's problem. Where exactly are you losing the packets?

This is the external interface of the machine that's directly connected to the modem. This happens with both machines connected directly to the modem, as well as my laptop when booted from a sysrescue CDROM.

I'm seeing lots of DNS query timeouts. The link is mostly idle, so I really don't think it's an issue with the kernel not being able to process the requests fast enough. It's also a Xeon E31240 @ 3.30GHz, so it's unlikely the computer isn't fast enough. The other server dropping packets is a Xeon E5-1650 v3 @ 3.50GHz.

The ISP is Optimum/Cablevision here in northern NJ.

business_kid · 06-07-2018, 04:15 AM

EDIT: Posted wrong earlier answer.

DNS timeout means that something in resolv.conf has either no DNS server running or is unreachable. Have you an internal DNS server? Otherwise it's an ISP problem.

gosssamer · 06-07-2018, 08:17 AM

Quote:

Originally Posted by business_kid

EDIT: Posted wrong earlier answer.

DNS timeout means that something in resolv.conf has either no DNS server running or is unreachable. Have you an internal DNS server? Otherwise it's an ISP problem.

Okay, I agree about the problem being with the ISP, but I need to be able to reproduce it. I believe the DNS problem is due to the packet loss, not a misconfiguration. It doesn't happen with every query. I don't believe a DNS query failure is enough, though.

Do you have any ideas on how I can reproduce the problem? Or perhaps how to more specifically identify the packets that are being dropped? Maybe with tcpdump or wireshark?

business_kid · 06-08-2018, 03:09 AM

Two good things to do would be to backup the logs, clear them, there try a reasonably large download & upload. Then write a script doing a nslookup on 20 or 30 hosts. Logs can be sent to the ISP.

gosssamer · 07-08-2018, 12:43 PM

Hi, I'm still having a problem and haven't been able to find a solution.

Can someone tell me if it's normal to experience packet loss on the interface connected to the cable modem?

# ifconfig br0
br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 68.195.193.42 netmask 255.255.255.248 broadcast 68.195.193.47
inet6 fe80::6c26:fdff:fe03:7f7e prefixlen 64 scopeid 0x20<link>
inet6 ::ec4:7aff:fea9:18de prefixlen 64 scopeid 0x0<global>
ether 0c:c4:7a:a9:18:de txqueuelen 500 (Ethernet)
RX packets 753480084 bytes 915628842230 (852.7 GiB)
RX errors 0 dropped 1690836 overruns 0 frame 0
TX packets 470183099 bytes 374063728411 (348.3 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

business_kid · 07-09-2018, 08:16 AM

Ok, a few things stand out to me. If the ISP was dropping packets with everyone, they would be put of business. Your connection must be on a 100Mb nic, sent out at 100Mb speeds. How is it throttled to 65Mb, and why? If it thinks 'I'm happy at 100Mb, so we'll talk at 100Mb', and has to be rudely reminded that the line isn't good for 100Mb, then yes, you would expect a few network errors as this happened. The modem should cache this and make it invisible to you.

You haven't mentioned actual tested speeds. Do a speed test. Morning is often better, before everyone gets into work but after night owls have stopped downloading. Schedule a speedtest for 5 A.M. It's still early in Europe then.

If you're forwarding the Gigabit network, why use a bridge instead of IP Forwarding? I thought the idea behind a bridge was to increase the output of a box beyond the output of the interfaces. Why bridge when you might be better forwarding? If you want to keep the rest of the gigabit LAN off the internet, you can use Virtual Interfaces to create a restricted connection on an otherwise open network. Bridging, IIRC, THE was invented in the dial up days. All network stuff is cheap anyhow so you could give them a separate network.

Lastly, does the ISP guarantee a 65 Mbit throughout, or do they (As in my case) offer 100 Mbit up the road, which drops to some extent on a piece of low specified cabling between them and me. I get about 40Mb on ethernet, and 35Mb (max) on WiFi. In this case, if the modem is caching, it would have the errors between it and ISP. What you show us is your problem between modem and box.

gosssamer · 07-09-2018, 09:25 AM

Quote:

Originally Posted by business_kid

Ok, a few things stand out to me. If the ISP was dropping packets with everyone, they would be put of business. Your connection must be on a 100Mb nic, sent out at 100Mb speeds. How is it throttled to 65Mb, and why? If it thinks 'I'm happy at 100Mb, so we'll talk at 100Mb', and has to be rudely reminded that the line isn't good for 100Mb, then yes, you would expect a few network errors as this happened. The modem should cache this and make it invisible to you.

The interface to the cable modem is 1000mbit, so I would think the flow control on the interfaces determine the proper speed.

Quote:

You haven't mentioned actual tested speeds. Do a speed test. Morning is often better, before everyone gets into work but after night owls have stopped downloading. Schedule a speedtest for 5 A.M. It's still early in Europe then.

I've tested it, and it's 65/15 with low ping times and no reported packet loss.

Quote:

If you're forwarding the Gigabit network, why use a bridge instead of IP Forwarding?

Because I'm using kvm to create virtual machines.

Quote:

Lastly, does the ISP guarantee a 65 Mbit throughout, or do they (As in my case) offer 100 Mbit up the road, which drops to some extent on a piece of low specified cabling between them and me. I get about 40Mb on ethernet, and 35Mb (max) on WiFi. In this case, if the modem is caching, it would have the errors between it and ISP. What you show us is your problem between modem and box.

Yes, the ISP claims they guarantee the 65mbit.

business_kid · 07-10-2018, 02:42 AM

Quote:

Originally Posted by gosssamer

Because I'm using kvm to create virtual machines.

So, the reality is many computers and VMs talking to a host which drops packets talking to the modem, or some other thing on the 1000 Mbit network, and everybody started with the wrong idea. I presume each VM runs a service - mail, or https or whatever. Is that correct? Why waste all our time withholding information?

Will you go back to your logs, and lspci outputs and identify:
1. If the VMs have physical or virtual network interfaces.
2. If the dropped packets have a common source, or come from all over the place.
3. As you will probably be sending us IPs, better print your routing table.
4.

If the dropped packets are from a common source, I'd suspect that source. If they haven't, it's something in or about your host.

gosssamer · 07-10-2018, 09:21 AM

Quote:

Originally Posted by business_kid

So, the reality is many computers and VMs talking to a host which drops packets talking to the modem, or some other thing on the 1000 Mbit network, and everybody started with the wrong idea. I presume each VM runs a service - mail, or https or whatever. Is that correct? Why waste all our time withholding information?

There's only one VM and it's seldom used. What bearing does a VM that is mostly idle have on this?

I also previously mentioned that there is another computer attached to this network with the same problem. It doesn't have any VMs.

Quote:

Will you go back to your logs, and lspci outputs and identify:

Which logs? I am not aware of any logs which are recording dropped packets. Are you sure you mean lspci? What does the computer device list have to do with this? Here's the ethernet device list:

Code:

# lspci|grep -i eth
06:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
06:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

Quote:

1. If the VMs have physical or virtual network interfaces.

Virtual. It's also showing dropped packets, however.

Code:

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 68.195.193.46  netmask 255.255.255.248  broadcast 68.195.193.47
        inet6 fe80::5054:ff:fe35:dd67  prefixlen 64  scopeid 0x20<link>
        inet6 ::5054:ff:fe35:dd67  prefixlen 64  scopeid 0x0<global>
        ether 52:54:00:35:dd:67  txqueuelen 1000  (Ethernet)
        RX packets 17779676  bytes 10500704612 (9.7 GiB)
        RX errors 0  dropped 587475  overruns 0  frame 0
        TX packets 14968367  bytes 3319127775 (3.0 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Quote:

2. If the dropped packets have a common source, or come from all over the place.

How can I determine this?

Quote:

3. As you will probably be sending us IPs, better print your routing table.

Code:

# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         68.195.193.41   0.0.0.0         UG    0      0        0 br0
68.195.193.40   0.0.0.0         255.255.255.248 U     0      0        0 br0
169.254.0.0     0.0.0.0         255.255.0.0     U     1003   0        0 eth1
169.254.0.0     0.0.0.0         255.255.0.0     U     1004   0        0 br0
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1
192.168.6.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1
192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0