Possible Network Issue?

XeNoMoRpH · 08-26-2009, 11:58 AM

Hello Everyone,

I recently have been tasked with figuring out why a machine running Fedora 6 has lost internet connectivity. We have several servers on one subnet (192.168.1.x) and another set of servers in the DMZ subnet (192.168.88.x). This server is in the DMZ and has no issue pinging any of the other servers (Windows machines) within the DMZ. However, that is where the communication stops. It cannot get to the other subnet like the other machines and cannot reach the outside (internet).

One test that we tried was changing the IP to one of the machines that did work to make sure it wasn't the router. Well, it still didn't work (I'm not necessarily satisfied with that test either). The network admin really thinks it is the machine.

Here is the ifconfig

Code:

eth0      Link encap:Ethernet  HWaddr 00:13:D4:0A:8B:1D
          inet addr:192.168.88.41  Bcast:192.168.88.255  Mask:255.255.255.0
          inet6 addr: fe80::213:d4ff:fe0a:8b1d/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:38324 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2593 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3987893 (3.8 MiB)  TX bytes:416339 (406.5 KiB)
          Interrupt:209 Base address:0xa800

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:3107 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3107 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:6812327 (6.4 MiB)  TX bytes:6812327 (6.4 MiB)

The hosts and nameservers are correct. I don't deal with linux, so I'm not really sure where to start.

I do know this only became a problem when the network was switched around. Two companies were sharing the same network, but we just recently split into two different networks. This is why I'm leaning towards the router more than anything else.

Tinkster · 08-26-2009, 12:11 PM

Hi, welcome to LQ!

The machine itself seems to have no problem on the physical
side of things. You may want to check whether it has host-
based firewalling enabled:

iptables -L

You mentioned that networks were "moved" ... did that machine
keep its IP, did it change?

What's its routing table?

route

Cheers,
Tink

XeNoMoRpH · 08-26-2009, 12:31 PM

Code:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination
RH-Firewall-1-INPUT  all  --  anywhere             anywhere

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
RH-Firewall-1-INPUT  all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain RH-Firewall-1-INPUT (2 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere
ACCEPT     icmp --  anywhere             anywhere            icmp any
ACCEPT     esp  --  anywhere             anywhere
ACCEPT     ah   --  anywhere             anywhere

ACCEPT     udp  --  anywhere             224.0.0.251         udp dpt:mdns
ACCEPT     udp  --  anywhere             anywhere            udp dpt:ipp
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ipp
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTAB                    LISHED
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:s                    sh
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:h                    ttp
REJECT     all  --  anywhere             anywhere            reject-with icmp-ho                    st-prohibited

I'll have to see if I can't find any routing tables. We have a Cisco 2811 Router and a Cisco PIX 501 Firewall.

Tinkster · 08-26-2009, 12:34 PM

Just issue the command 'route" on the linux box.

ivanmacx · 08-26-2009, 12:42 PM

For the routing table just enter

Code:

route -n

You could also post the results of

Code:

cat /etc/hosts.allow
cat /etc/hosts.deny

XeNoMoRpH · 08-26-2009, 12:48 PM

route -n

Code:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.88.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth0
0.0.0.0         192.168.88.1    0.0.0.0         UG    0      0        0 eth0

cat /etc/hosts.allow (just has comments)

Code:

#
# hosts.allow   This file describes the names of the hosts which are
#               allowed to use the local INET services, as decided
#               by the '/usr/sbin/tcpd' server.
#

cat /etc/hosts.deny (just has comments as well)

Code:

#
# hosts.deny    This file describes the names of the hosts which are
#               *not* allowed to use the local INET services, as decided
#               by the '/usr/sbin/tcpd' server.
#
# The portmap line is redundant, but it is left to remind you that
# the new secure portmap uses hosts.deny and hosts.allow.  In particular
# you should know that NFS uses portmap!

Tinkster · 08-26-2009, 12:58 PM

Hmmm ... nothing untoward or unexpected. I'll dare say
the problem sits somewhere else. I've seen interesting
cases where e.g., one windows machine had a private subnet
on a secondary interface that happened to use the same
address range that a Linux box was on on the corporate
LAN ... took a while to find out just what was happening
and prodding the windows and network guys hard enough
to start looking ;}

XeNoMoRpH · 08-26-2009, 01:03 PM

I forgot to mention that the IP of the machine did not change after the split. I'll have to dig for some more information to really learn what happened with the split and how it was originally.

XeNoMoRpH · 08-26-2009, 01:23 PM

I decided to check the message log and found that the machine uses Avahi. My initial search showed a lot of people having issues with this. Here is the log. It has done this 3-4 times today. Not sure if it's right.

Code:

Aug 26 13:37:57 db avahi-daemon[2661]: Withdrawing address record for 192.168.88.41 on eth0.
Aug 26 13:37:57 db avahi-daemon[2661]: Leaving mDNS multicast group on interface eth0.IPv4 with addr
ess 192.168.88.41.
Aug 26 13:37:57 db avahi-daemon[2661]: iface.c: interface_mdns_mcast_join() called but no local addr
ess available.
Aug 26 13:37:57 db avahi-daemon[2661]: Interface eth0.IPv4 no longer relevant for mDNS.
Aug 26 13:37:57 db NetworkManager: <information>        SWITCH: terminating current connection 'eth0
' because it's no longer valid.
Aug 26 13:37:57 db NetworkManager: <information>        Deactivating device eth0.
Aug 26 13:37:57 db avahi-daemon[2661]: Interface eth0.IPv6 no longer relevant for mDNS.
Aug 26 13:37:57 db avahi-daemon[2661]: Withdrawing address record for fe80::213:d4ff:fe0a:8b1d on et
h0.
Aug 26 13:37:57 db NetworkManager: nm_device_is_802_3_ethernet: assertion `dev != NULL' failed
Aug 26 13:37:57 db NetworkManager: nm_device_is_802_11_wireless: assertion `dev != NULL' failed
Aug 26 13:37:57 db avahi-daemon[2661]: New relevant interface eth0.IPv6 for mDNS.
Aug 26 13:37:57 db avahi-daemon[2661]: Joining mDNS multicast group on interface eth0.IPv6 with addr
ess fe80::213:d4ff:fe0a:8b1d.
Aug 26 13:37:57 db avahi-daemon[2661]: IPV6_ADD_MEMBERSHIP failed: Invalid argument
Aug 26 13:37:57 db avahi-daemon[2661]: Registering new address record for fe80::213:d4ff:fe0a:8b1d o
n eth0.
Aug 26 13:37:57 db dhcdbd: message_handler: message handler not found under /com/redhat/dhcp/eth0 fo
r sub-path eth0.dbus.get.reason
Aug 26 13:37:57 db NetworkManager: <information>        Will activate wired connection 'eth0' becaus
e it now has a link.
Aug 26 13:37:57 db NetworkManager: <information>        SWITCH: no current connection, found better
connection 'eth0'.
Aug 26 13:37:57 db dhcdbd: message_handler: message handler not found under /com/redhat/dhcp/eth0 fo
r sub-path eth0.dbus.get.reason
Aug 26 13:37:57 db NetworkManager: <information>        Will activate connection 'eth0'.
Aug 26 13:37:57 db NetworkManager: <information>        Device eth0 activation scheduled...
Aug 26 13:37:57 db NetworkManager: <information>        Activation (eth0) started...
Aug 26 13:37:57 db NetworkManager: <information>        Activation (eth0) Stage 1 of 5 (Device Prepa
re) scheduled...
Aug 26 13:37:57 db NetworkManager: <information>        Activation (eth0) Stage 1 of 5 (Device Pre
re) started...
Aug 26 13:37:57 db NetworkManager: <information>        Activation (eth0) Stage 2 of 5 (Device Con
gure) scheduled...
Aug 26 13:37:57 db NetworkManager: <information>        Activation (eth0) Stage 1 of 5 (Device Pre
re) complete.
Aug 26 13:37:57 db NetworkManager: <information>        Activation (eth0) Stage 2 of 5 (Device Con
gure) starting...
Aug 26 13:37:57 db NetworkManager: <information>        Activation (eth0) Stage 2 of 5 (Device Con
gure) successful.
Aug 26 13:37:57 db NetworkManager: <information>        Activation (eth0) Stage 3 of 5 (IP Configu
 Start) scheduled.
Aug 26 13:37:57 db NetworkManager: <information>        Activation (eth0) Stage 2 of 5 (Device Con
gure) complete.
Aug 26 13:37:57 db NetworkManager: <information>        Activation (eth0) Stage 3 of 5 (IP Configu
 Start) started...
Aug 26 13:37:57 db NetworkManager: <information>        Activation (eth0) Stage 4 of 5 (IP Configu
 Get) scheduled...
Aug 26 13:37:57 db NetworkManager: <information>        Activation (eth0) Stage 3 of 5 (IP Configu
 Start) complete.
Aug 26 13:37:57 db NetworkManager: <information>        Activation (eth0) Stage 4 of 5 (IP Configu
 Get) started...
Aug 26 13:37:57 db NetworkManager: <information>        Activation (eth0) Stage 5 of 5 (IP Configu
 Commit) scheduled...
Aug 26 13:37:57 db NetworkManager: <information>        Activation (eth0) Stage 4 of 5 (IP Configu
 Get) complete.
Aug 26 13:37:57 db NetworkManager: <information>        Activation (eth0) Stage 5 of 5 (IP Configu
 Commit) started...
Aug 26 13:37:57 db avahi-daemon[2661]: New relevant interface eth0.IPv4 for mDNS.
Aug 26 13:37:57 db avahi-daemon[2661]: Joining mDNS multicast group on interface eth0.IPv4 with ad
ess 192.168.88.41.
Aug 26 13:37:57 db avahi-daemon[2661]: Registering new address record for 192.168.88.41 on eth0.
Aug 26 13:37:58 db NetworkManager: <information>        Activation (eth0) successful, device activ
ed.
Aug 26 13:37:58 db NetworkManager: <information>        Activation (eth0) Finish handler scheduled
Aug 26 13:37:58 db NetworkManager: <information>        Activation (eth0) Stage 5 of 5 (IP Configu
 Commit) complete.
Aug 26 13:37:59 db kernel: eth0: Media Link On 100mbps full-duplex
Aug 26 13:38:37 db avahi-daemon[2661]: Invalid query packet.

XeNoMoRpH · 08-27-2009, 11:58 AM

Okay, a little update. I put the server on the internal switch and it worked just fine. It could ping hosts outside and talk to all of the servers on both subnets. I switched it back and of course it's not happy. I'm not sure why the DMZ is an issue. I'm leaning more towards an issue with the switch, minus the test with switching the IP to something known to work.

Tinkster · 08-27-2009, 12:32 PM

Has the machine been moved to another port on the switch/router?

ivanmacx · 08-28-2009, 12:35 PM

It sounds like an issue with the router between the two subnets, as you say. Does that have any MAC address filters in operation perhaps?

XeNoMoRpH · 08-31-2009, 10:15 AM

Okay, here is an update. I ended up adding the subnet 192.168.1.0 into the routing table. So now it looks like this:

Code:

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.88.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0
0.0.0.0         192.168.88.1    0.0.0.0         UG    0      0        0 eth0

I can now ping machines within that subnet. I also fixed a DNS issue. It couldn't access the outside DNS address, so changing it to the internal one fixed that. Now the problem is it still can't access the outside. The computer knows all of the hosts and IPs, it just can't get through.

ivanmacx · 09-03-2009, 07:30 AM

It really sounds like an issue with the router or the firewall. I would get your network folk to check that there's no specific block or filter relating to that machine.

jschiwal · 09-03-2009, 07:51 AM

Code:

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.88.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0
0.0.0.0         192.168.88.1    0.0.0.0         UG    0      0        0 eth0

Since the server has an IP address of 192.168.88.41, don't you need a gateway entry when the target is on the 192.168.1.0 subnet?

For the first route output, the default gateway at 192.168.88.1 would be used.