Hi,

I'm looking for diagnostic ideas. My situation:
Ethernet (PCIe, builtin igb) network packets arrive corrupted/changed (ICMP requests -- viewed in wireshark), OS doesn't process them at all. Corruption is not random, but I haven't been able to find a pattern.

What I tried:
Disable all firewalls.
Boot liveCD.
Change cable.
remove all switches from path. Still happens on p2p link.
Add another PCIe ethernet card (r8169), corruption is the exactly the same.

Chipset: X570. CPU: AMD 5800X

At this point I'm thinking the PCIe bus is doing weird stuff. Any ideas what more to try?

You're on the right track swapping things out. Here's some ideas.

• Can you get the packets correctly on another machine?
• Can you swap out any of the cabling?
• Can you slow speeds down in your box?
• Can you (even temporarily) pick the signals up at the earliest point?
• Can you do a thorough nic <--> nic test of your box?

You're beaten only when you run out of ideas. Try as many as you can (and guess who might have done some faultfinding before)

See if checksum offload is set on nic.

Cheksum offload, off:
Still same behaviour.

ICMP request example dump:
byte 0x33 is changed from 0xd5 to 0xc5, byte 0x4e from 0x24 to 0x04.
0x4e is consistently wrong. 0x33 only sometimes (but it isn't constant at the source). I haven't checked the rest too thoroughly...

Another interesting behaviour is that without a running egress ping (no replies received) I don't get ingress requests (corrupted).

This is driving me insane. When it started: When I logged into a desktop session, and plasma-nm applet took over. Before that NetworkManager managed the connection fine. Since then it doesn't work even without networkmanager, or under a liveCD.

All networking material has been either replaced or removed until a single cable remained, and that swapped out too. Even the other peer has been changed. Still exactly the same behaviour.
I don't think I can do better than wireshark at each end of the cable. Will try to find another machine to act as bridge.

I haven't seen it mentioned yet, but some other ideas to run by the mental checklist:
- Check cable rating vs link negotiation speed. Sometimes folks will have an old Cat5 cable hanging around and try to use it with a Gigabit Ethernet connection. It should be Cat5e or better.
- How long is your cable connection? Probably not the issue, but Ethernet is rated for 100m.
- Also, do you have any sources of interference near the cable? Powerful motors, high voltage, ... sources of electrical or magnetic interference.
- Tried a USB Ethernet adapter?

I would work at eliminating things with successful tests, not negative ones.

What I mean is: A second/different network card also throwing errors doesn't mean the first one is good. But a network card doing successful 100 metre transfers means the network card is good.

That said, have you changed out the long trunk of network cable, even by laying something along the floor? BTW, false ceilings are great for temporary cabling.

Hi,

Thanks everybody for suggestions. Time being scarce, and contiguous time intervals being even scarcer I didn't manage to sit down and continue with this for a while. However, I now have the culprit.

My PCIe bus is indeed shot.
If you look at this Zen 3 interconnect diagram, All PCIe connected through the x570 chipset has the same behaviour. Using the GPU lanes for the network card results in a working network.
For some reason, SATA ports seem to work OK. (btrfs doesn't complain about checksums)
An even stronger proof is that a part of the AM4 socket plastic is partially melted under the CPU. Something terrible happened, even though the PC is behind a surge-protecting power socket. It's a miracle this machine still somehow works.

So, mystery "solved", one for the books...

Edit: as a bonus, a screenshot of wireshark upon reception of a packet consisting only of dashes "-", corruption nicely visible.

Attached Thumbnails

 

In all circumstances, the silicon dies before the chip socket melts. There's over 50º C at least in the difference. So the socket melting looks like sabotage, or outside thermal forces. Very strange…

Anyhow, replace your m/b and mark this solved.