How do I detect/diagnose a NIC suspected of "coming and going"?

RichardBronosky · 06-25-2008, 12:15 AM

I have a machine that I suspect may have a hardware problem. Do you have any suggestions on how to detect that the card is loosing connection for tiny fractions of a second? I don't know if the switch, the nic, the cable, or either of the RJ45 connectors are bad. I'd love to hear something like, "That's easy just set this logging on and every time the nic does a handshake it will get logged."

Please no suggestions like, "well, why not try to ping it continuously".

Let me be clear, this server (MySQL) is under HIGH net traffic and a few times a day, the app servers log that the connection was lost. It happens for a few connections NOT a few minutes or even seconds, connections (like milliseconds).

I will reply with an lspci & uname in the AM.

Thanks in advance!

tajamari · 06-25-2008, 12:36 AM

[QUOTE=

Please no suggestions like, "well, why not try to ping it continuously".

[/QUOTE]

Well thats the basic part. Do it. If there'a a timeout, check if there are collisions on the switch, on your server you can check the interface if there are errors on RX and TX packets.

nolinuxnollife · 06-25-2008, 01:17 AM

As your mysql error says "connection loss" , this includes a diagnoses of all tcp layers

1)Chek mysql permissions from server to client

2) chek whether some unauthorised users are trying to connect to mysql , this is because if i type "mysql -h your_mysql_host -u root -p" , i will be able to establish a connection , but unless i authenticate i will not have access to DB..this will be logged in server logs

3)The version of db connectivity drivers used app servers (you have not mentioned), might be an issue

4)the maximum number of connections that mysql is able to bear (if its unable to handle more client requests try adding one more NIC to the server box)

u will get more if u tryout above said

farslayer · 06-25-2008, 07:15 AM

if it's the NIC you wish to look at then just look at ifconfig

Quote:

user@it-etch:~/$ /sbin/ifconfig

eth0 Link encap:Ethernet HWaddr 00:1a:b2:78:87:n8
inet addr:192.168.0.4 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe::::08:6708/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:22273329 errors:0 dropped:0 overruns:0 frame:0
TX packets:12121744 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1643791792 (1.5 GiB) TX bytes:983919359 (938.3 MiB)
Interrupt:209

If there's a problem with the NIC or even in the physical layer it will show up in the errors section of the output.
if the carrier number is climbing then that means the interface is bouncing, the link pulse has been lost and restored similar to unplugging the cable, and plugging it back in. This is on by default there is no need to enable logging to see these numbers.

On a decent managed switch you can also view the switchport statistics to gather similar information.

If they are all ZERO as in my output then you need to look elsewhere for your connection issues, possibly at mysql as suggested .

RichardBronosky · 06-25-2008, 09:11 AM

Quote:

Originally Posted by farslayer

if it's the NIC you wish to look at then just look at ifconfig
...
If they are all ZERO as in my output then you need to look elsewhere for your connection issues, possibly at mysql as suggested .

That's the kind of thing I needed, thanks! Unfortunately, it seems that is not catching issues even when I artificially create them. I tried "watch -n1 ifconfig" while pulling the ethernet cable several times on my sandbox. The only thing that changed was the /[RT]X/ numbers.

Any other ideas?

Here is the additional info I promised.

Code:

$ cat /etc/redhat-release ; uname -r ; ifconfig|sed '/addr:/d' ; lspci
Red Hat Enterprise Linux AS release 4 (Nahant Update 6)
2.6.9-67.0.4.ELsmp
eth0      Link encap:Ethernet  HWaddr 00:30:48:5B:7E:8A  
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:956922713 errors:0 dropped:0 overruns:0 frame:0
          TX packets:146882150 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:64298981893 (59.8 GiB)  TX bytes:214535012200 (199.8 GiB)
          Interrupt:185 

eth1      Link encap:Ethernet  HWaddr 00:30:48:5B:7E:8B  
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:5627925488 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7112933139 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:876165294028 (815.9 GiB)  TX bytes:6533312647637 (5.9 TiB)
          Interrupt:193 

lo        Link encap:Local Loopback  
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:4826198 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4826198 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:2637572778 (2.4 GiB)  TX bytes:2637572778 (2.4 GiB)

00:06.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8111 PCI (rev 07)
00:07.0 ISA bridge: Advanced Micro Devices [AMD] AMD-8111 LPC (rev 05)
00:07.1 IDE interface: Advanced Micro Devices [AMD] AMD-8111 IDE (rev 03)
00:07.2 SMBus: Advanced Micro Devices [AMD] AMD-8111 SMBus 2.0 (rev 02)
00:07.3 Bridge: Advanced Micro Devices [AMD] AMD-8111 ACPI (rev 05)
00:0a.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 13)
00:0a.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC (rev 01)
00:0b.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 13)
00:0b.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC (rev 01)
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
01:01.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID (rev 01)
02:05.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 10)
02:05.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 10)
03:00.0 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b)
03:00.1 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b)
03:04.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)