LinuxQuestions.org
Visit the LQ Articles and Editorials section
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices

Reply
 
Search this Thread
Old 03-15-2004, 09:45 PM   #1
jimlaur
LQ Newbie
 
Registered: Jul 2003
Location: Wilmington, DE
Distribution: Debian
Posts: 6

Rep: Reputation: 0
do these symptoms mean my system is compromised?


Perhaps I am too paranoid, but I started to investigate some odd counts in my
firewall rules by adding a log rule, and what I saw both puzzles and alarms me.

The log is showing icmp destination unreachable messages arriving that say
that my host tried to contact another machine (that I never heard of) and failed.

The reported outgoing connection attempt is to a udp port that my firewall should not
allow to pass for a new connection, usually 1026 or 1027.

I do not know what to make of this.

I do not think that someone is launching packets from somewhere else but using
my IP address because the messages are logged by a connection tracking firewall
rule that tests for ESTABLISHED,RELATED packets, which means, I think, that
the attempt did come from my machine.

Is it possible that a web page could execute a script that would attempt this
connection, bypassing the NEW connection filter that restricts outgoing ports
because it is classified as RELATED?

Or, is this a sign of some malicious code running in the kernel on my machine?

The machine is running RH9 Linux with iptables.

What would happen if the connection succeeded?
Should I be worried, or is this just noise?

What other information would be helpful to understand this problem?

I do not know how to even classify this problem so that I can search for related
messages on the web.

If this is a well known problem, perhaps someone can tell me how to describe it
so I can search for info on related things.

If this is not the right place to ask for help, perhaps someone will direct me
to a better place.

Thanks.

Jim
 
Old 03-15-2004, 09:51 PM   #2
verdeboy2k
Member
 
Registered: Jan 2004
Location: /dev/random
Distribution: Gentoo amd64, CrunchBang amd64
Posts: 350

Rep: Reputation: 32
I do not know exactly what the problem is, but you could run a packet sniffer protocol on the affected machine(s) and see if it catches any suspicious packets. This would also allow you to see what the packets contained. Also, have you installed any new software recently that might access the 'net?
 
Old 03-15-2004, 10:12 PM   #3
Capt_Caveman
Senior Member
 
Registered: Mar 2003
Distribution: Fedora
Posts: 3,658

Rep: Reputation: 57
It might be a bit more clear if you posted the exact iptables rule(s) that you have logging those packets and the associated log messages. Also, it's not clear to me whether you are getting log messages about icmp dest unreachable packets or outgoing udp packets or both. One thing about logging packets: it can be extremely useful if positioned correctly, but it can generate a whole boat-load of noise if it's not.
 
Old 03-15-2004, 10:28 PM   #4
jimlaur
LQ Newbie
 
Registered: Jul 2003
Location: Wilmington, DE
Distribution: Debian
Posts: 6

Original Poster
Rep: Reputation: 0
more info

Thank you for your interest and help.

I wondered about the sniffer.
I guess I will have to learn how to use that next.

Here are some more details from the log and the firewall.

This is a typical log entry:

Mar 15 17:34:24 kernel: IPT ICON IN=eth0 OUT= MAC=* SRC=209.178.xxx.yyy DST=66.92.aaa.bbb LEN=112 TOS=0x00 PREC=0x00 TTL=243 ID=44282 DF PROTO=ICMP TYPE=3 CODE=3 [SRC=66.92.aaa.bbb DST=209.178.xxx.yyy LEN=836 TOS=0x00 PREC=0x00 TTL=117 ID=10219 PROTO=UDP SPT=18108 DPT=1026 LEN=816 ]

(I took out the mac address and suppressed parts of the IP addresses
I think there is enough to make it clear.)

Here is the rule that produced it:

$IPT -A INPUT -i $INT_IF -m conntrack --ctstate ESTABLISHED,RELATED -j LOG --log-prefix "IPT ICON "

( $INT_IF is the ethernet port facing the internet )

The immediately preceding rule uses state tracking, and does not trigger!:

$IPT -A INPUT -p icmp -m state --state ESTABLISHED,RELATED -j ACCEPT

I put in the conntack rule to see if I was missing something.
(The original rule is the same except for -j ACCEPT instead of LOG)
I stated to get a few counts, so I added the log rule.
When I saw the icmp messages, I began to worry.
 
Old 03-16-2004, 11:40 PM   #5
Capt_Caveman
Senior Member
 
Registered: Mar 2003
Distribution: Fedora
Posts: 3,658

Rep: Reputation: 57
In most cases I would say port 1026 UDP packets with forged IPs are a dead ringer for the Windows Messenger SPAM that is all over the internet nowadays. But if the conntrack rule is matching, then the ICMPs *should* be in response to an actual connection attempt originating from your machine that is sitting in the state table. I'm still not entirely convinced though. Look at the ttl values on the icmp packet. The returned packet supposedly coming from your machine has a ttl which is not even close to the linux default of 64. Hopefully someone will correct me if I'm wrong, but that seems screwy to me.

Definitely weird. Does it seem to happen consistantly like at a certain time (like on startup) or with certain applications (like when you use the internet, etc). It would definitely be informative to try the sniffer idea like verdeboy2k suggested, but try and capture some of the outbound packets destined for the 209.178 address rather than the ICMP packets. Give ethereal a try, it's a little more user friendly.
 
Old 03-17-2004, 02:47 PM   #6
jimlaur
LQ Newbie
 
Registered: Jul 2003
Location: Wilmington, DE
Distribution: Debian
Posts: 6

Original Poster
Rep: Reputation: 0
more info, things look better

Hi Captain,

Thanks for the follow up.
I have been gathering more information too.
I rewrote the firewall rules and turned on tcpdump to trace traffic to/from udp 1026 & 1027.
I think I have covered the possibilities now, please see the details below.

I am less worried today because I think I can convince you, below, that there are no packets coming from my machine that could prompt those destination unreachable messages. I am further reassured by your observation that the original packets do not look like Linux packets.

I agree it is definitely weird that the iptables conntrack module is picking up what seem like packets returned to a spoofed ip address. My working hypothesis is that either there is something I do not understand about using the conntrack module or else that the tracking rules it applies are too loose. As a first step I am splitting the firewall log rule between ESTABLISHED and RELATED so that I can see which condition it thinks applies.

I am going to pursue the matter because I think it could be a weakness if the module accepts icmp packets that are not really associated with an established or related connection. The rule I (naively) wrote does not specify a protocol, and, without proof to the contrary, I assume that accepting packets from other protocols could open a vulnerability, and I assume the module maintainers will be concerned too.

I was hoping that someone else had seen this problem before and I could avoid doing this work. If this is really a new observation, I hope you will help me to describe what we learn in a way that will help find this information in the future.

I hope you agree that given that I could not tell if the icmp messages meant that my system was compromised or that the conntrack module had a problem, going to a security forum first was the right thing to do. I guess I do have to learn how to look for bug reports, that is where I guess I should go next, now that I know what part of the code I suspect.

Here are some of the details of what I have found:

First, you are right about the incoming udp packets.
These are clearly spam, and obnoxious spam at that.
I think I have seen some pop-ups in mozilla that warn me
that my windows machine has a vulnerability.
In fact, I have turned off unrequested pop-ups.
So I guess mozilla may listen on one or these ports,
or maybe these particular packets are directed at some IM application,
that I, luckily, do not run.

Here is one line from the tcpdump report:

06:06:27.150595 61.17.xxx.yyy.777 > 66.92.aaa.bbb.1026: [no cksum] udp 1285 (DF) (ttl 106, id 0, len 1313)

And here is the corresponding log record:

Mar 17 06:06:27 kernel: IPT UBI IN=eth0 OUT= MAC=* SRC=61.17.xxx.yyy DST=66.92.aaa.bbb LEN=1313 TOS=0x00 PREC=0x00 TTL=106 ID=0 DF PROTO=UDP SPT=777 DPT=1026 LEN=1293

Finally, here is text fragment from the packet:

"Your system is affected, download the patch from the address below ! "

(yeah, right!)

Anyway, the point is that the firewall is logging the new packets coming in,
and tcpdump is tracing all upd packets to or from 1026 or 1027.
Here is the tcpdump command I used:

tcpdump -i eth0 -p -s0 -w funny-packets2 udp dst port 1026 or udp dst port 1027

OK, so during the time that tcpdump was running, all the upd packets to or from 1026 or 1027 were all incoming, like the above example, there were no outgoing udp packets.

Almost all these udp packets matched the -m state --state NEW rule.
I can match the log and tcpdump entries except at two timestamps. In these cases tcpdump recorded 3 identical packets and the firewall only logged one. I think this is still ok.

Also, in the firewall, I added checks for packets being accepted in or out on udp ports 1026 and 1027 under the -m state --state ESTABLISHED,RELATED rule in the forward, input and output chains. All the udp packets are accounted for, and no udp packets were sent or received with a destination port of 1026 or 1027 through these state rules.

During this time, when there were only incoming packets on upd 1026 and 1027, there were several more instances of the icmp message saying my machine had failed to reach ....
This is the critical step. I got the destination unreachable error messages during a time when I believe I had not sent any such packets.

I think this makes a good case that these icmp messages are about packets with spoofed ip addresses that did not originate on my machine. I think to be super sure I would have to run a packet sniffer on a separate host on the same network, but I think I will go look for information about the iptables conntrack module first.

I hope you agree with my analysis.
I will keep listening to this thread, and I will post any more relevant information.
Please let me know if you think there is anything else I should do.
Thank you for your support.

Jim

Last edited by jimlaur; 03-17-2004 at 04:17 PM.
 
Old 03-18-2004, 03:17 AM   #7
Skunk_Face
Member
 
Registered: Jan 2004
Posts: 54

Rep: Reputation: 15
sorry peeps...dont really have an anwser here...just that this thread has aroused my curiosity.

firstly, what correlation does a UDP packet on port 1026/1027 have with an icmp packet on the same port???

and I was kinda under the impression that both UDP and ICMP are somewhat not connection oriented protocols so how does connection tracking work with state RELATED or ESTABLISHED for that matter?
 
Old 03-18-2004, 08:54 AM   #8
jimlaur
LQ Newbie
 
Registered: Jul 2003
Location: Wilmington, DE
Distribution: Debian
Posts: 6

Original Poster
Rep: Reputation: 0
Well, I am not an expert, for sure, I am searching for that answer now. I got into this by adding a rule to my firewall to see what would happen. I had a rule with the pattern -p icmp -m state --state ESTABLISHED,RELATED . I then added a following rule -m conntrack --ctstate ESTABLISHED,RELATED. I saw I was getting a few matches, so I added a log rule. When I saw what the icmp messages implied, I started to worry.

So far it seems clear that icmp messages can be matched to existing connections, and that udp messages are entered into the connection table even though they are "stateless". It seems that for the icmp destination unreachable message to match the pattern there would have to be an entry in the connection table to match to. However, this module seems fairly complicated, and I may be misusing it out of ignorance. It may be that it is matching some other connection or because the icmp messages are repeating or for some other reason I do not understand. I am trying to go throught the documentation at http://www.netfilter.org/ to learn more. I think the next thing is to try to find a mailing list where I can discuss this with someone more knowledgeable on the operations of this code.
 
Old 03-18-2004, 08:59 AM   #9
verdeboy2k
Member
 
Registered: Jan 2004
Location: /dev/random
Distribution: Gentoo amd64, CrunchBang amd64
Posts: 350

Rep: Reputation: 32
The spoofed IP address ICMP packets do sound like someone was port-scanning you and those were the only ones that weren't bounced by the firewall.
 
Old 03-18-2004, 09:34 AM   #10
jimlaur
LQ Newbie
 
Registered: Jul 2003
Location: Wilmington, DE
Distribution: Debian
Posts: 6

Original Poster
Rep: Reputation: 0
I am skeptical of the idea that someone was port scanning me. It seems like a lot of work to generate phony icmp packets containing information about another packet I was supposed to have sent. Also, sometimes the icmp messages do not come from the same host as the unreachable udp packet was addressed to. That is, sometimes the icmp message is from an apparent gateway of some sort telling me that a host on the gateways network is unreachable (so the source in the icmp packet and the destination in the udp packet are different, but similar). My first concern is to be sure that my machine is not, in fact, trying to send those udp packets to anyone else without my knowledge. If my machine did not send the original udp packet then I do not yet see how sending me an icmp packet about it is going to tell them anything about my ports. But I could easily be missing something, so fill me in if I am, please.
 
Old 03-18-2004, 12:20 PM   #11
Capt_Caveman
Senior Member
 
Registered: Mar 2003
Distribution: Fedora
Posts: 3,658

Rep: Reputation: 57
I would agree that the icmp packets are not likely to be forged. Rather the initial udp packet sent from some unknown host to the 209.178.xxx.yyy host is the one more likely to be forged. I would guess that the initial udp packet sent to port 1026 contained Windows Messenger spam and was forged to have jim's IP address. When the remote gateway machine couldn't reach the destination host, it sent an ICMP host unreachable response to the source IP address on the packet. Because that IP was forged, the ICMP packet was sent to Jim's machine instead of the true source (the spammer). While the port 1026 udp packets captured by tcpdump were the same kind of Windows messenger spam, those packets are likely un-related to the initial packet that generated the ICMP packet. In general ICMP scanning is not a very efficient means of information gathering. If one were to go through the effort of crafting specialty packets, there are a lot better ways of doing it, though it still could be possible.

What is curious to me is that the conntrack ctstate rule matched the packet. From my understanding both conntrack and state should handle the ESTABLISHED,RELATED matches identically, in that they should both check the connection state table and should only match if an existing entry is in the table waiting for a reply of some form or another. If you look at the iptables man pages and other documentation (which is sparse at best), the description of the conntrack and state matches are essentially identical with regard to NEW, INVALID, RELATED, and ESTABLISHED states (in fact they are verbatim in the man pages). The only difference I can determine is that with conntrack ctstate, you can access additional information in the state table, but that seems to only give you the added ability to match packets based on DNAT/SNAT information. In this situation, that should not have any impact. The fact that you are not capturing outgoing port 1026 udp packets, would suggest that you shouldn't have any waiting connections in the state table or alternatively that the system has been compromised (which I highly doubt). You can visually verify the state table by looking in /proc/net/ip_conntrack and seeing if you have any waiting udp connections. It might be a good idea to post the question to the iptables mailing list and seeing if anyone has observed this behaviour before. It would appear that the conntrack match is busted, but to be honest I'm not very familiar with the inner workings of the conntrack module, so this could be a feature(?).

Note that the concept of "state" here is differnet from the networking concept of "states" (as in UDP being a "stateless" protocol). In iptables-speak, you can have UDP/ICMP connection states. The iptables term is dependent on whether you have sent a packet and are waiting for a response versus the networking concepts of packet sequence numbering and SYN/ACK/RST flags.

If you're interested on reading up, the best docs I've seen on iptables states and state matching is available through the netfilter site or here:

http://www.sns.ias.edu/~jns/security...conntrack.html
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
System possibly compromised kloppster Linux - Security 7 07-12-2004 03:30 PM
high speed system compromised witeshark Linux - Security 3 04-14-2004 03:53 PM
System compromised BruceCadieux Linux - Security 20 09-29-2003 08:24 PM
System compromised? Comatose51 Linux - Security 3 07-11-2003 08:28 AM
Help: I think my system has been compromised! Comatose51 Linux - General 2 06-29-2003 05:00 PM


All times are GMT -5. The time now is 08:14 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration