[SOLVED] server not working anymore with local network... please, please HELP!

masavini · 07-12-2014, 04:53 AM

hi,
please forgive this long post, but i'm facing a huge problem with my server...

1) since a couple of weeks ago, i was getting a lot of spam by many contacts of mine. i suspected someone was hacking my server, so i installed and run clamav. a few threats were found in firefox cache, but nothing really serious. after a few days spam ceased, so i simply forgot about it...

2) a couple of days ago i set up a brand new access point in the warehouse. 2 powerline adapters bring lan over there, then the access point spreads a wifi signal. it seemed to work pretty fine, but...
i was in the office, working on my laptop (connected with wifi). i opened a ssh session on the server, then i moved the laptop in the warehouse.
internet connection was fine, but ssh session was stuck. i gave no importance to that, thinking it was related to some temporary connection bug.
i went back to the office, tried to reconnect to server and ssh did not come up still.
i rebooted both server and laptop, but i still could not connect to server with local network. i could connect to the server with the public ip using ssh, remmina and http, but not using local network.

now i can connect from one laptop to another, but none of them can connect to server or vice versa. network printers (attached to the server) do not work as well, nor server's samba shares.
looks like the server is not connected to local network, but it is!

Code:

laptop:/tmp $ ssh server
ssh: connect to host server port 22: No route to host

Code:

server:/tmp $ ssh laptop
ssh: connect to host laptop port 22: Connection timed out

i spent half of the night trying to debug this...

first i booted an old server backup (without clamav), but problems still occurs.
then i checked /var/log/auth.log, and found a LOT of these strings:
"Address 192.168.1.4 maps to server, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!"

i found a lot of documentation about this issue, and tried the 2 most common solutions:
- adding "UseDNS no", in /etc/ssh/sshd_config
- removing ~/.ssh/known_hosts
but none of them seemed to fix anything...

do you think my server has been hacked and used (maybe even now) to spam? the only modification i've done on my system, by now, is to disable ssh port forwading to server on the router... so no one can remotely connect to it...

ANY suggestion would be much appreciated...

dorian33 · 07-12-2014, 05:41 AM

1. POSSIBLE BREAK-IN ATTEMPT is just a warning; if you know who use this address (maybe it is even your local IP) is is really doesn't matter

2. which address you want to connect running 'ssh server' command?
in other words which address the host 'server' should be ?

and which address the name 'server' is resolved?
just run 'ping server' agains 'ssh server' and you will see which is the IP of the 'server'

btw: what is responsible for IP resolution for the name 'server' ?

3. are you able to run 'ssh <local_ip_of_the_server>' ?

up to know everything looks like you have name rosolving problems rather than ssh (and other services) issues.

masavini · 07-12-2014, 07:52 AM

Quote:

Originally Posted by dorian33

1. POSSIBLE BREAK-IN ATTEMPT is just a warning; if you know who use this address (maybe it is even your local IP) is is really doesn't matter

2. which address you want to connect running 'ssh server' command?
in other words which address the host 'server' should be ?

and which address the name 'server' is resolved?
just run 'ping server' agains 'ssh server' and you will see which is the IP of the 'server'

btw: what is responsible for IP resolution for the name 'server' ?

3. are you able to run 'ssh <local_ip_of_the_server>' ?

up to know everything looks like you have name rosolving problems rather than ssh (and other services) issues.

thank you for your fast reply...
1-2) yes, i know that ip (192.168.1.4): it's the one used by server on eth0 (static ip)

Code:

laptop:/tmp $ cat /etc/hosts
127.0.0.1	localhost.localdomain	localhost
::1	laptop	localhost6.localdomain6	localhost6
127.0.1.1	laptop

192.168.1.4     server	server.local

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

3) no, i get the same result as runnning ssh server:

Code:

laptop:/tmp $ ssh 192.168.1.4
ssh: connect to host 192.168.1.4 port 22: No route to host

masavini · 07-12-2014, 08:15 AM

mmm, it's getting interesting...

Code:

laptop:/tmp $ nmap laptop -Pn

Starting Nmap 6.40 ( http://nmap.org ) at 2014-07-12 15:04 CEST
Nmap scan report for laptop (127.0.0.1)
Host is up (0.00036s latency).
Other addresses for laptop (not scanned): 127.0.1.1
rDNS record for 127.0.0.1: localhost.localdomain
Not shown: 991 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
25/tcp   open  smtp
80/tcp   open  http
139/tcp  open  netbios-ssn
445/tcp  open  microsoft-ds
631/tcp  open  ipp
3306/tcp open  mysql
5800/tcp open  vnc-http
8080/tcp open  http-proxy

Nmap done: 1 IP address (1 host up) scanned in 0.05 seconds

if i check laptop's ports from server:

Code:

server:/tmp $ nmap laptop -Pn

Starting Nmap 6.40 ( http://nmap.org ) at 2014-07-12 15:09 CEST
Nmap scan report for laptop (192.168.1.3)
Host is up.
All 1000 scanned ports on laptop (192.168.1.3) are filtered

Nmap done: 1 IP address (1 host up) scanned in 201.30 seconds

how can it be? man-in-the-middle?

dorian33 · 07-12-2014, 08:58 AM

"No route to host" - what is the routing table?
maybe default gw is not set...

masavini · 07-12-2014, 09:26 AM

more tests...

i attached the laptop to a lan cable.
now i can ssh into mostro and vice versa.

if i unplug the cable and go with the wifi, the problem occurs again.

this, however, can't be a laptop wifi misconfiguration, since the same problem occurs even with other laptops and smartphones: no one can connect to server using the wifi...

masavini · 07-12-2014, 09:29 AM

Quote:

Originally Posted by dorian33

"No route to host" - what is the routing table?
maybe default gw is not set...

default gateway is 192.168.1.1, the router ip...

dorian33 · 07-12-2014, 10:38 AM

both at hosts with IP 192.168.1.3 (laptop)and 192.168.1.4 (server)
- check firewall
- check arp table

jpollard · 07-12-2014, 11:17 AM

192.168.x.x addresses are nonroutable.

Thus the wireless AP (being a router) will not route such target addresses.

It is also likely that the AP is a NAT, and thus hides the machines using the AP for internet access.

masavini · 07-12-2014, 11:31 AM

Quote:

Originally Posted by dorian33

both at hosts with IP 192.168.1.3 (laptop)and 192.168.1.4 (server)
- check firewall
- check arp table

both server and laptop have same firewall configuration:

Code:

$ sudo iptables -L -n
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

arp tables:

Code:

laptop:/tmp $ sudo arp
Address                  HWtype  HWaddress           Flags Mask            Iface
192.168.1.1              ether   98:fc:11:cf:d4:79   C                     wlan0
server                           (incomplete)                              wlan0

Code:

server:/tmp $ sudo arp
Address                  HWtype  HWaddress           Flags Mask            Iface
192.168.1.1              ether   98:fc:11:cf:d4:79   C                     eth1
192.168.1.129            ether   c8:df:7c:7c:6e:0f   C                     eth1
192.168.1.2              ether   60:a4:4c:0a:8a:9e   C                     eth1
laptop                   ether   c4:85:08:96:94:98   C                     eth1

there is one host i don't know: 192.168.1.129... who is it? it's not even listed in the list of connected clients on the router...
one more weird thing is the fact that 192.168.1.2 is laptop's eth0 static ip (the mac is right), but it is not connected right now, so i can't tell why it's listed there...

masavini · 07-12-2014, 11:39 AM

Quote:

Originally Posted by jpollard

192.168.x.x addresses are nonroutable.

Thus the wireless AP (being a router) will not route such target addresses.

It is also likely that the AP is a NAT, and thus hides the machines using the AP for internet access.

really sorry, but i can't understand what you mean... the wireless AP (d-link wl2100ap) is unplugged since yesterday... how can it affect actual behaviour of wireless network?

dorian33 · 07-12-2014, 12:15 PM

I belive

Code:

server                           (incomplete)                              wlan0

explains the problem

No entry in arp table makes impossible to send eth frame.
I suggest to use arping 192.168.1.4 and repeat arp table check.

I see that laptop is using wlan0 interface so it probably communicates via WiFi as wlanX is typically the name of WiFi i/f.
But I don't know where the AP (or WiFi router) which laptop connects to is connected and know nothing about the AP/WiFi router config so I believe I can't help more.
Really difficult to make up the reasons...

Maybe laptop MAC is blocked at AP/router?

Regarding 192.168.1.129 - it looks like it is a Nokia mobile phone client since according to this page the MAC is Nokia's one.

As far as stupid arp entry 192.168.1.2 - just clear the cache and check if it appears again.

jpollard · 07-12-2014, 12:16 PM

If you are not using a wireless AP, then check the network topology.

Routers will not route 192.168.x.x addresses period. What they WILL do is provide NAT services - thus the network visible to the outside is the address of the router, not the addresses behind the router.

SOME routers can be specifically configured to route the private addresses, but it gets tricky to set up, and is error prone (public IP numbers don't have the same problem).

Depending on the router, it might be possible to define the router as a "bridge" allowing private addresses on one side to talk to private addresses on the other side, but again, this is a special configuration.

These private addresses were specifically designated for testing purposes and not production (at least originally, before NAT procedures were added).

masavini · 07-12-2014, 01:04 PM

network topology is VERY simple:

a wireless modem router (linksys x2000 - 192.168.1.1) connected to the server (lan cable - 192.168.1.4).
several laptops and mobiles (yes, 192,168.1.129 is a nokia mobile) connected with wifi.

router can ping laptop (192.168.1.3) with no issue.

i tried what dorian33 suggested:

Code:

laptop:/tmp $ arping 192.168.1.4 -I wlan0
ARPING 192.168.1.4 from 192.168.1.3 wlan0
^CSent 53 probes (53 broadcast(s))
Received 0 response(s)
laptop:/tmp $ arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
192.168.1.1              ether   98:fc:11:cf:d4:79   C                     wlan0
192.168.1.4                      (incomplete)

Code:

server:/tmp $ arping 192.168.1.3 -I eth1 
ARPING 192.168.1.3 from 192.168.1.4 eth1
Unicast reply from 192.168.1.3 [C4:85:08:96:94:98]  20.534ms
^CSent 24 probes (1 broadcast(s))
Received 1 response(s)

masavini · 07-12-2014, 01:10 PM

i tried and manually fix server arp entry, but it did not fix anything...

Code:

laptop:/tmp $ sudo arp -s 192.168.1.4 20:CF:30:19:B5:CE
laptop:/tmp $ arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
192.168.1.1              ether   98:fc:11:cf:d4:79   C                     wlan0
192.168.1.4              ether   20:cf:30:19:b5:ce   CM                    wlan0
laptop:/tmp $ ssh server
^C
laptop:/tmp $

i'm getting VERY frustrated...