How can I track the cause of this...

I've recently been provided with a Technicolor 582n router. Not a very impressive bit of hardware, but I'm stuck with it for a while. Since I wasn't very confident in the quality of its firewall, I've set it to log to the syslog on my server.

I've got port 22 open to allow me SSH access from the internet (set to use keys only), so I'm a bit wary of possible breaches. I've also set up dynamic dns server (using no-ip.com) so I can find the system if its IP changes.

Looking through the router logs, I see quite a large number of attempted port scans. Generally they seem fairly harmless. However, in the last few days I've been seeing something that does make me suspicious, since its coming from within my own network. Here is a sample of the router log to illustrate (192.168.1.192 is a desktop machine running Salix 14.0):

Info    Jul 19 16:00:31 SNTP Synchronised to server: 5.2.16.145
Error   Jul 19 15:57:40 FIREWALL replay check (1 of 1): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 117.83.111.179 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 15:55:52 FIREWALL icmp check (1 of 30): Protocol: ICMP Src ip: 218.75.155.248 Dst ip: 2.103.28.90 Type: Echo Reply Code: 0
Info    Jul 19 15:55:52 IDS fragment parser : fragment out-of-order (1 of 8) : 218.75.155.248 2.103.28.90 0028 ICMP frag 15244:8@1472+
Error   Jul 19 15:46:35 FIREWALL replay check (1 of 1): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 2.60.144.27 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 15:44:13 FIREWALL replay check (1 of 3): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 2.60.144.27 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 15:22:11 FIREWALL replay check (1 of 2): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 112.21.68.98 Type: Destination Unreachable Code: Port Unreacheable
Info    Jul 19 15:00:31 SNTP Synchronised to server: 94.228.40.3
Error   Jul 19 14:23:46 FIREWALL replay check (1 of 2): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 110.114.185.208 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 14:04:41 FIREWALL replay check (1 of 3): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 183.83.4.10 Type: Destination Unreachable Code: Port Unreacheable
Info    Jul 19 14:00:31 SNTP Synchronised to server: 217.114.59.66
Error   Jul 19 13:59:55 FIREWALL replay check (1 of 2): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 183.83.4.10 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 13:58:11 FIREWALL replay check (1 of 2): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 183.83.4.10 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 13:47:36 FIREWALL replay check (1 of 1): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 212.225.195.213 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 13:45:36 FIREWALL replay check (1 of 1): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 183.2.100.30 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 13:43:12 FIREWALL replay check (1 of 2): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 95.59.77.193 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 13:11:41 FIREWALL replay check (1 of 2): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 10.160.157.125 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 13:10:40 FIREWALL replay check (1 of 2): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 60.167.240.184 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 13:09:03 FIREWALL replay check (1 of 3): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 87.253.47.146 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 13:04:19 FIREWALL replay check (1 of 2): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 87.253.47.146 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 13:02:03 FIREWALL replay check (1 of 1): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 87.253.47.146 Type: Destination Unreachable Code: Port Unreacheable
Info    Jul 19 13:00:31 SNTP Synchronised to server: 193.238.80.20

So far as I can see, there should be no reason for 192.168.1.192 to be generating these "FIREWALL replay check" entries. Until a couple of days ago, I'd never seen one. Whatever the cause, it is coming from the base system - I rebooted the machine from remote last night and even though no-one was logged in, found a few more of these entries when I checked the logs on getting home.

Can anyone tell me what might cause this type of error, or how best to go tracking it down? I've installed and run rkhunter - it hasn't reported any problems, but it was only installed after the problem started, so I can't be sure that its results are 100%.

Most of the incoming probes seem to originate in China, though I can't imagine that there is anything on my network to interest anyone.

While I try and find the cause, I've left that machine off the network, and looking at the logs I'm only seeing incoming probes, nothing going back out from my machines.

I'd be grateful for any suggestions as to the best method of tracking this. So far, my searching hasn't given me much information as to what is going on....

Paul.

Originally Posted by toothandnail So far as I can see, there should be no reason for 192.168.1.192 to be generating these "FIREWALL replay check" entries. Until a couple of days ago, I'd never seen one. (..) Can anyone tell me what might cause this type of error, or how best to go tracking it down?

First of all, and unfortunately this is a reflex seen in novice and seasoned users alike, the best way ever to thwart any analysis is to reboot a machine: doing that neatly erases all volatile user, process, open files and network connection information. Secondly ICMP is the "Internet Control Message Protocol", mostly used to correct errors wrt transmission or routing. To analyze what ICMP messages a system sends it's easier to set up a few rules in the machines firewall than to rely in your Thomsons logs IMHO because iptables can verbosely log what messages get sent according to their relation. Most likely you'll find these messages get sent in response to halted services (for example any file sharing apps), as "ghosts" from connections the previous user of the IP had (if your ISP recycles dynamic IP addresses quickly) or port scans.

Originally Posted by unSpawn First of all, and unfortunately this is a reflex seen in novice and seasoned users alike, the best way ever to thwart any analysis is to reboot a machine: doing that neatly erases all volatile user, process, open files and network connection information. Secondly ICMP is the "Internet Control Message Protocol", mostly used to correct errors wrt transmission or routing. To analyze what ICMP messages a system sends it's easier to set up a few rules in the machines firewall than to rely in your Thomsons logs IMHO because iptables can verbosely log what messages get sent according to their relation. Most likely you'll find these messages get sent in response to halted services (for example any file sharing apps), as "ghosts" from connections the previous user of the IP had (if your ISP recycles dynamic IP addresses quickly) or port scans.

Sorry to be slow - too many 12 hour shifts at the moment.

I used ps and lsof to see if I could spot anything suspicious on 192.168.1.192 before I restarted it. Apart from a long list of consolekit entries in the process list (not sure why, but I've seen them before with Salix, so didn't think they were suspicious), I didn't see anything that looked out of place (well, to me anyway). Also didn't find any strange open files.

Reason I rebooted the machine was I wondered if there might be some process left over from a couple of torrents that I had running for 24 hours or so before the entries stated. Doesn't look like it, since similar entries started again shortly after I restarted the machine. The things that make me suspicious are the fact that, other than something left from the torrents, I couldn't see any reason one of my boxes would be sending packets to the IPs involved. It also seemed odd that only one of the machines on my local network would be doing this. There are no entries at all due to traffic from my server, which is Slackware-based and provides file storage, plus DHCP and DNS servers.

Since then, things have got even stranger. I was having a few problems with Salix, so I decided to have a play with something else - I can easily put Salix back later if I feel a need. So I installed the latest SolydX 64-bit version. While I didn't format my home partition, I created a new user and renamed the old user's home directory. Within 30 minutes of completing the new installation, I started seeing similar log entries. The root partition was reformatted by the installation, but I don't know whether the swap partition was, so maybe there is something there that might be triggering the problem.

I'm not familiar enough with these sorts of things to know where to start in even assessing whether this is a threat or not. I'm also not familiar enough with setting up Linux firewalls to know how best to set something up on the machine to monitor from there. I certainly agree that the router isn't an ideal place to try. Not only is there too little information in its logging, there is also too little explanation of what the log entries mean - even finding a decent user manual for the thing has proved almost impossible.

I guess I'm going to need a crash course in setting up a firewall and other monitoring to get to the bottom of this. I'd be grateful for any suggestions as to where best to start...

Paul.

Run 'iptables-save > /tmp/iptables.txt' as root, attach "/tmp/iptables.txt" as plain text?

Originally Posted by unSpawn Run 'iptables-save > /tmp/iptables.txt' as root, attach "/tmp/iptables.txt" as plain text?

SolydX comes with ufw, though its disabled by default. I enabled it, then ran iptables-save.

Results are here:

As I said before ICMP is used to correct errors. At the bottom of your UFW rule set you'll see a

-A ufw-user-limit -j REJECT --reject-with icmp-port-unreachable

rule with which it rejects all requests for new connections (regardless of it being TCP or UDP) with a "port unreachable" ICMP message. Without firewall running, if a remote host would try to set up a new connection, you'd see outbound "reset" messages if the connection was TCP, but since you're seeing "port unreachable" ICMP replies this means they were UDP requests. IIRC Bittorrent only uses outbound connections from client (you) to remote trackers but between peers the majority of traffic will be UDP-based. If you're not satisfied with this answer you should decide if you want to learn the UFW syntax (I try to be as distro-agnostic as possible but theres limits ;-p) or "plain" iptables. The latter may look awkward at first but it's actually quite easy to set up a default rule set requiring ninety per cent less rules than UFW...

Originally Posted by toothandnail Here is a sample of the router log to illustrate (192.168.1.192 is a desktop machine running Salix 14.0): Info Jul 19 16:00:31 SNTP Synchronised to server: 5.2.16.145 Error Jul 19 15:57:40 FIREWALL replay check (1 of 1): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 117.83.111.179 Type: Destination Unreachable Code: Port Unreacheable Error Jul 19 15:55:52 FIREWALL icmp check (1 of 30): Protocol: ICMP Src ip: 218.75.155.248 Dst ip: 2.103.28.90 Type: Echo Reply Code: 0 Info Jul 19 15:55:52 IDS fragment parser : fragment out-of-order (1 of 8) : 218.75.155.248 2.103.28.90 0028 ICMP frag 15244:8@1472+ Error Jul 19 15:46:35 FIREWALL replay check (1 of 1): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 2.60.144.27 Type: Destination Unreachable Code: Port Unreacheable Error Jul 19 15:44:13 FIREWALL replay check (1 of 3): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 2.60.144.27 Type: Destination Unreachable Code: Port Unreacheable Error Jul 19 15:22:11 FIREWALL replay check (1 of 2): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 112.21.68.98 Type: Destination Unreachable Code: Port Unreacheable Info Jul 19 15:00:31 SNTP Synchronised to server: 94.228.40.3 Error Jul 19 14:23:46 FIREWALL replay check (1 of 2): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 110.114.185.208 Type: Destination Unreachable Code: Port Unreacheable Error Jul 19 14:04:41 FIREWALL replay check (1 of 3): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 183.83.4.10 Type: Destination Unreachable Code: Port Unreacheable Info Jul 19 14:00:31 SNTP Synchronised to server: 217.114.59.66 Error Jul 19 13:59:55 FIREWALL replay check (1 of 2): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 183.83.4.10 Type: Destination Unreachable Code: Port Unreacheable Error Jul 19 13:58:11 FIREWALL replay check (1 of 2): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 183.83.4.10 Type: Destination Unreachable Code: Port Unreacheable Error Jul 19 13:47:36 FIREWALL replay check (1 of 1): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 212.225.195.213 Type: Destination Unreachable Code: Port Unreacheable Error Jul 19 13:45:36 FIREWALL replay check (1 of 1): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 183.2.100.30 Type: Destination Unreachable Code: Port Unreacheable Error Jul 19 13:43:12 FIREWALL replay check (1 of 2): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 95.59.77.193 Type: Destination Unreachable Code: Port Unreacheable Error Jul 19 13:11:41 FIREWALL replay check (1 of 2): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 10.160.157.125 Type: Destination Unreachable Code: Port Unreacheable Error Jul 19 13:10:40 FIREWALL replay check (1 of 2): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 60.167.240.184 Type: Destination Unreachable Code: Port Unreacheable Error Jul 19 13:09:03 FIREWALL replay check (1 of 3): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 87.253.47.146 Type: Destination Unreachable Code: Port Unreacheable Error Jul 19 13:04:19 FIREWALL replay check (1 of 2): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 87.253.47.146 Type: Destination Unreachable Code: Port Unreacheable Error Jul 19 13:02:03 FIREWALL replay check (1 of 1): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 87.253.47.146 Type: Destination Unreachable Code: Port Unreacheable Info Jul 19 13:00:31 SNTP Synchronised to server: 193.238.80.20 So far as I can see, there should be no reason for 192.168.1.192 to be generating these "FIREWALL replay check" entries. Until a couple of days ago, I'd never seen one.

This is an ICMP type 3, code 3 packet ("port unreachable"). Your PC, 192.168.1.192, is sending these messages to various external hosts because these hosts have attempted to contact a TCP or UDP socket to which no service is currently bound.

The question you need to ask yourself is how hosts on the public Internet are able to send packets directly to a PC on your internal network, one that even has a non-routeble IP address. There could be a number of reasons why this is happening:

1. you may have forwarded a port in your router to the wrong internal IP address, or chosen the wrong internal and/or external port number, and now your Salic PC receives packages meant for another host and/or socket
2. you may have forwarded a port in your router to the right IP address and port, but the server application is not currently running on your Salix PC
3. the router may have the Universal Plug and Play feature enabled, and some software on your PC may have opened the port
4. you have been running a P2P application of some sort on the Salix PC, and the messages appear whenever you close the application, as external hosts participating in the P2P network are still trying to reach it
5. your router is one of many erroneously exposing its UPnP services to the Internet, and someone on the Internet is trying to take advantage of that

Looking at the log, the messages themselves are probably caused by a P2P app, or it's a poor attempt at stealth port scanning, as one of the address is in fact an RFC 1918 address (10.160.157.125).

You should check the port forwarding and UPnP settings in your router. If nothing obvious turns up, reset the router and reflash the firmware if you can.