LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices

Reply
 
Search this Thread
Old 07-19-2013, 08:30 PM   #1
toothandnail
Member
 
Registered: Apr 2007
Location: Oxfordshire, UK
Distribution: Arch, SolydX, Salix64
Posts: 74

Rep: Reputation: 5
How can I track the cause of this...


I've recently been provided with a Technicolor 582n router. Not a very impressive bit of hardware, but I'm stuck with it for a while. Since I wasn't very confident in the quality of its firewall, I've set it to log to the syslog on my server.

I've got port 22 open to allow me SSH access from the internet (set to use keys only), so I'm a bit wary of possible breaches. I've also set up dynamic dns server (using no-ip.com) so I can find the system if its IP changes.

Looking through the router logs, I see quite a large number of attempted port scans. Generally they seem fairly harmless. However, in the last few days I've been seeing something that does make me suspicious, since its coming from within my own network. Here is a sample of the router log to illustrate (192.168.1.192 is a desktop machine running Salix 14.0):

Code:
Info    Jul 19 16:00:31 SNTP Synchronised to server: 5.2.16.145
Error   Jul 19 15:57:40 FIREWALL replay check (1 of 1): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 117.83.111.179 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 15:55:52 FIREWALL icmp check (1 of 30): Protocol: ICMP Src ip: 218.75.155.248 Dst ip: 2.103.28.90 Type: Echo Reply Code: 0
Info    Jul 19 15:55:52 IDS fragment parser : fragment out-of-order (1 of 8) : 218.75.155.248 2.103.28.90 0028 ICMP frag 15244:8@1472+
Error   Jul 19 15:46:35 FIREWALL replay check (1 of 1): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 2.60.144.27 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 15:44:13 FIREWALL replay check (1 of 3): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 2.60.144.27 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 15:22:11 FIREWALL replay check (1 of 2): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 112.21.68.98 Type: Destination Unreachable Code: Port Unreacheable
Info    Jul 19 15:00:31 SNTP Synchronised to server: 94.228.40.3
Error   Jul 19 14:23:46 FIREWALL replay check (1 of 2): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 110.114.185.208 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 14:04:41 FIREWALL replay check (1 of 3): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 183.83.4.10 Type: Destination Unreachable Code: Port Unreacheable
Info    Jul 19 14:00:31 SNTP Synchronised to server: 217.114.59.66
Error   Jul 19 13:59:55 FIREWALL replay check (1 of 2): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 183.83.4.10 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 13:58:11 FIREWALL replay check (1 of 2): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 183.83.4.10 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 13:47:36 FIREWALL replay check (1 of 1): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 212.225.195.213 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 13:45:36 FIREWALL replay check (1 of 1): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 183.2.100.30 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 13:43:12 FIREWALL replay check (1 of 2): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 95.59.77.193 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 13:11:41 FIREWALL replay check (1 of 2): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 10.160.157.125 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 13:10:40 FIREWALL replay check (1 of 2): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 60.167.240.184 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 13:09:03 FIREWALL replay check (1 of 3): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 87.253.47.146 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 13:04:19 FIREWALL replay check (1 of 2): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 87.253.47.146 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 13:02:03 FIREWALL replay check (1 of 1): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 87.253.47.146 Type: Destination Unreachable Code: Port Unreacheable
Info    Jul 19 13:00:31 SNTP Synchronised to server: 193.238.80.20
So far as I can see, there should be no reason for 192.168.1.192 to be generating these "FIREWALL replay check" entries. Until a couple of days ago, I'd never seen one. Whatever the cause, it is coming from the base system - I rebooted the machine from remote last night and even though no-one was logged in, found a few more of these entries when I checked the logs on getting home.

Can anyone tell me what might cause this type of error, or how best to go tracking it down? I've installed and run rkhunter - it hasn't reported any problems, but it was only installed after the problem started, so I can't be sure that its results are 100%.

Most of the incoming probes seem to originate in China, though I can't imagine that there is anything on my network to interest anyone.

While I try and find the cause, I've left that machine off the network, and looking at the logs I'm only seeing incoming probes, nothing going back out from my machines.

I'd be grateful for any suggestions as to the best method of tracking this. So far, my searching hasn't given me much information as to what is going on....

Paul.
 
Old 07-20-2013, 05:07 AM   #2
unSpawn
Moderator
 
Registered: May 2001
Posts: 26,943
Blog Entries: 54

Rep: Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731
Quote:
Originally Posted by toothandnail View Post
So far as I can see, there should be no reason for 192.168.1.192 to be generating these "FIREWALL replay check" entries. Until a couple of days ago, I'd never seen one. (..) Can anyone tell me what might cause this type of error, or how best to go tracking it down?
First of all, and unfortunately this is a reflex seen in novice and seasoned users alike, the best way ever to thwart any analysis is to reboot a machine: doing that neatly erases all volatile user, process, open files and network connection information. Secondly ICMP is the "Internet Control Message Protocol", mostly used to correct errors wrt transmission or routing. To analyze what ICMP messages a system sends it's easier to set up a few rules in the machines firewall than to rely in your Thomsons logs IMHO because iptables can verbosely log what messages get sent according to their relation. Most likely you'll find these messages get sent in response to halted services (for example any file sharing apps), as "ghosts" from connections the previous user of the IP had (if your ISP recycles dynamic IP addresses quickly) or port scans.
 
1 members found this post helpful.
Old 07-21-2013, 08:22 PM   #3
toothandnail
Member
 
Registered: Apr 2007
Location: Oxfordshire, UK
Distribution: Arch, SolydX, Salix64
Posts: 74

Original Poster
Rep: Reputation: 5
Quote:
Originally Posted by unSpawn View Post
First of all, and unfortunately this is a reflex seen in novice and seasoned users alike, the best way ever to thwart any analysis is to reboot a machine: doing that neatly erases all volatile user, process, open files and network connection information. Secondly ICMP is the "Internet Control Message Protocol", mostly used to correct errors wrt transmission or routing. To analyze what ICMP messages a system sends it's easier to set up a few rules in the machines firewall than to rely in your Thomsons logs IMHO because iptables can verbosely log what messages get sent according to their relation. Most likely you'll find these messages get sent in response to halted services (for example any file sharing apps), as "ghosts" from connections the previous user of the IP had (if your ISP recycles dynamic IP addresses quickly) or port scans.
Sorry to be slow - too many 12 hour shifts at the moment.

I used ps and lsof to see if I could spot anything suspicious on 192.168.1.192 before I restarted it. Apart from a long list of consolekit entries in the process list (not sure why, but I've seen them before with Salix, so didn't think they were suspicious), I didn't see anything that looked out of place (well, to me anyway). Also didn't find any strange open files.

Reason I rebooted the machine was I wondered if there might be some process left over from a couple of torrents that I had running for 24 hours or so before the entries stated. Doesn't look like it, since similar entries started again shortly after I restarted the machine. The things that make me suspicious are the fact that, other than something left from the torrents, I couldn't see any reason one of my boxes would be sending packets to the IPs involved. It also seemed odd that only one of the machines on my local network would be doing this. There are no entries at all due to traffic from my server, which is Slackware-based and provides file storage, plus DHCP and DNS servers.

Since then, things have got even stranger. I was having a few problems with Salix, so I decided to have a play with something else - I can easily put Salix back later if I feel a need. So I installed the latest SolydX 64-bit version. While I didn't format my home partition, I created a new user and renamed the old user's home directory. Within 30 minutes of completing the new installation, I started seeing similar log entries. The root partition was reformatted by the installation, but I don't know whether the swap partition was, so maybe there is something there that might be triggering the problem.

I'm not familiar enough with these sorts of things to know where to start in even assessing whether this is a threat or not. I'm also not familiar enough with setting up Linux firewalls to know how best to set something up on the machine to monitor from there. I certainly agree that the router isn't an ideal place to try. Not only is there too little information in its logging, there is also too little explanation of what the log entries mean - even finding a decent user manual for the thing has proved almost impossible.

I guess I'm going to need a crash course in setting up a firewall and other monitoring to get to the bottom of this. I'd be grateful for any suggestions as to where best to start...

Paul.
 
Old 07-22-2013, 01:29 AM   #4
unSpawn
Moderator
 
Registered: May 2001
Posts: 26,943
Blog Entries: 54

Rep: Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731
Run 'iptables-save > /tmp/iptables.txt' as root, attach "/tmp/iptables.txt" as plain text?
 
Old 07-24-2013, 08:02 AM   #5
toothandnail
Member
 
Registered: Apr 2007
Location: Oxfordshire, UK
Distribution: Arch, SolydX, Salix64
Posts: 74

Original Poster
Rep: Reputation: 5
Quote:
Originally Posted by unSpawn View Post
Run 'iptables-save > /tmp/iptables.txt' as root, attach "/tmp/iptables.txt" as plain text?

SolydX comes with ufw, though its disabled by default. I enabled it, then ran iptables-save.

Results are here:
Attached Files
File Type: txt iptables.txt (4.9 KB, 10 views)
 
Old 07-24-2013, 01:36 PM   #6
unSpawn
Moderator
 
Registered: May 2001
Posts: 26,943
Blog Entries: 54

Rep: Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731
As I said before ICMP is used to correct errors. At the bottom of your UFW rule set you'll see a
Code:
-A ufw-user-limit -j REJECT --reject-with icmp-port-unreachable
rule with which it rejects all requests for new connections (regardless of it being TCP or UDP) with a "port unreachable" ICMP message. Without firewall running, if a remote host would try to set up a new connection, you'd see outbound "reset" messages if the connection was TCP, but since you're seeing "port unreachable" ICMP replies this means they were UDP requests. IIRC Bittorrent only uses outbound connections from client (you) to remote trackers but between peers the majority of traffic will be UDP-based. If you're not satisfied with this answer you should decide if you want to learn the UFW syntax (I try to be as distro-agnostic as possible but theres limits ;-p) or "plain" iptables. The latter may look awkward at first but it's actually quite easy to set up a default rule set requiring ninety per cent less rules than UFW...
 
Old 07-24-2013, 02:02 PM   #7
Ser Olmy
Senior Member
 
Registered: Jan 2012
Distribution: Slackware
Posts: 1,910

Rep: Reputation: Disabled
Quote:
Originally Posted by toothandnail View Post
Here is a sample of the router log to illustrate (192.168.1.192 is a desktop machine running Salix 14.0):

Code:
Info    Jul 19 16:00:31 SNTP Synchronised to server: 5.2.16.145
Error   Jul 19 15:57:40 FIREWALL replay check (1 of 1): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 117.83.111.179 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 15:55:52 FIREWALL icmp check (1 of 30): Protocol: ICMP Src ip: 218.75.155.248 Dst ip: 2.103.28.90 Type: Echo Reply Code: 0
Info    Jul 19 15:55:52 IDS fragment parser : fragment out-of-order (1 of 8) : 218.75.155.248 2.103.28.90 0028 ICMP frag 15244:8@1472+
Error   Jul 19 15:46:35 FIREWALL replay check (1 of 1): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 2.60.144.27 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 15:44:13 FIREWALL replay check (1 of 3): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 2.60.144.27 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 15:22:11 FIREWALL replay check (1 of 2): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 112.21.68.98 Type: Destination Unreachable Code: Port Unreacheable
Info    Jul 19 15:00:31 SNTP Synchronised to server: 94.228.40.3
Error   Jul 19 14:23:46 FIREWALL replay check (1 of 2): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 110.114.185.208 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 14:04:41 FIREWALL replay check (1 of 3): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 183.83.4.10 Type: Destination Unreachable Code: Port Unreacheable
Info    Jul 19 14:00:31 SNTP Synchronised to server: 217.114.59.66
Error   Jul 19 13:59:55 FIREWALL replay check (1 of 2): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 183.83.4.10 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 13:58:11 FIREWALL replay check (1 of 2): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 183.83.4.10 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 13:47:36 FIREWALL replay check (1 of 1): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 212.225.195.213 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 13:45:36 FIREWALL replay check (1 of 1): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 183.2.100.30 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 13:43:12 FIREWALL replay check (1 of 2): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 95.59.77.193 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 13:11:41 FIREWALL replay check (1 of 2): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 10.160.157.125 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 13:10:40 FIREWALL replay check (1 of 2): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 60.167.240.184 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 13:09:03 FIREWALL replay check (1 of 3): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 87.253.47.146 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 13:04:19 FIREWALL replay check (1 of 2): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 87.253.47.146 Type: Destination Unreachable Code: Port Unreacheable
Error   Jul 19 13:02:03 FIREWALL replay check (1 of 1): Protocol: ICMP Src ip: 192.168.1.192 Dst ip: 87.253.47.146 Type: Destination Unreachable Code: Port Unreacheable
Info    Jul 19 13:00:31 SNTP Synchronised to server: 193.238.80.20
So far as I can see, there should be no reason for 192.168.1.192 to be generating these "FIREWALL replay check" entries. Until a couple of days ago, I'd never seen one.
This is an ICMP type 3, code 3 packet ("port unreachable"). Your PC, 192.168.1.192, is sending these messages to various external hosts because these hosts have attempted to contact a TCP or UDP socket to which no service is currently bound.

The question you need to ask yourself is how hosts on the public Internet are able to send packets directly to a PC on your internal network, one that even has a non-routeble IP address. There could be a number of reasons why this is happening:
  1. you may have forwarded a port in your router to the wrong internal IP address, or chosen the wrong internal and/or external port number, and now your Salic PC receives packages meant for another host and/or socket
  2. you may have forwarded a port in your router to the right IP address and port, but the server application is not currently running on your Salix PC
  3. the router may have the Universal Plug and Play feature enabled, and some software on your PC may have opened the port
  4. you have been running a P2P application of some sort on the Salix PC, and the messages appear whenever you close the application, as external hosts participating in the P2P network are still trying to reach it
  5. your router is one of many erroneously exposing its UPnP services to the Internet, and someone on the Internet is trying to take advantage of that
Looking at the log, the messages themselves are probably caused by a P2P app, or it's a poor attempt at stealth port scanning, as one of the address is in fact an RFC 1918 address (10.160.157.125).

You should check the port forwarding and UPnP settings in your router. If nothing obvious turns up, reset the router and reflash the firmware if you can.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Track DDOS SamiMukahhal Linux - Security 3 10-07-2012 01:40 PM
LXer: Track Me! Just Track Me, GNOME Project! LXer Syndicated Linux News 0 03-02-2011 01:41 AM
LXer: Go track yourself LXer Syndicated Linux News 0 10-11-2009 11:41 PM
What is the Best track to be an RHCE? toraif Linux - Certification 6 06-30-2009 09:55 PM
does anyone know how to rip a cd as one track? solusrex Linux - Software 1 03-27-2006 06:12 AM


All times are GMT -5. The time now is 11:17 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration