Linux - SecurityThis forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Introduction to Linux - A Hands on Guide
This guide was created as an overview of the Linux Operating System, geared toward new users as an exploration tour and getting started guide, with exercises at the end of each chapter.
For more advanced trainees it can be a desktop reference, and a collection of the base knowledge needed to proceed with system and network administration. This book contains many real life examples derived from the author's experience as a Linux system and network administrator, trainer and consultant. They hope these examples will help you to get a better understanding of the Linux system and that you feel encouraged to try out things on your own.
Click Here to receive this Complete Guide absolutely free.
Perhaps I am too paranoid, but I started to investigate some odd counts in my
firewall rules by adding a log rule, and what I saw both puzzles and alarms me.
The log is showing icmp destination unreachable messages arriving that say
that my host tried to contact another machine (that I never heard of) and failed.
The reported outgoing connection attempt is to a udp port that my firewall should not
allow to pass for a new connection, usually 1026 or 1027.
I do not know what to make of this.
I do not think that someone is launching packets from somewhere else but using
my IP address because the messages are logged by a connection tracking firewall
rule that tests for ESTABLISHED,RELATED packets, which means, I think, that
the attempt did come from my machine.
Is it possible that a web page could execute a script that would attempt this
connection, bypassing the NEW connection filter that restricts outgoing ports
because it is classified as RELATED?
Or, is this a sign of some malicious code running in the kernel on my machine?
The machine is running RH9 Linux with iptables.
What would happen if the connection succeeded?
Should I be worried, or is this just noise?
What other information would be helpful to understand this problem?
I do not know how to even classify this problem so that I can search for related
messages on the web.
If this is a well known problem, perhaps someone can tell me how to describe it
so I can search for info on related things.
If this is not the right place to ask for help, perhaps someone will direct me
to a better place.
I do not know exactly what the problem is, but you could run a packet sniffer protocol on the affected machine(s) and see if it catches any suspicious packets. This would also allow you to see what the packets contained. Also, have you installed any new software recently that might access the 'net?
It might be a bit more clear if you posted the exact iptables rule(s) that you have logging those packets and the associated log messages. Also, it's not clear to me whether you are getting log messages about icmp dest unreachable packets or outgoing udp packets or both. One thing about logging packets: it can be extremely useful if positioned correctly, but it can generate a whole boat-load of noise if it's not.
( $INT_IF is the ethernet port facing the internet )
The immediately preceding rule uses state tracking, and does not trigger!:
$IPT -A INPUT -p icmp -m state --state ESTABLISHED,RELATED -j ACCEPT
I put in the conntack rule to see if I was missing something.
(The original rule is the same except for -j ACCEPT instead of LOG)
I stated to get a few counts, so I added the log rule.
When I saw the icmp messages, I began to worry.
In most cases I would say port 1026 UDP packets with forged IPs are a dead ringer for the Windows Messenger SPAM that is all over the internet nowadays. But if the conntrack rule is matching, then the ICMPs *should* be in response to an actual connection attempt originating from your machine that is sitting in the state table. I'm still not entirely convinced though. Look at the ttl values on the icmp packet. The returned packet supposedly coming from your machine has a ttl which is not even close to the linux default of 64. Hopefully someone will correct me if I'm wrong, but that seems screwy to me.
Definitely weird. Does it seem to happen consistantly like at a certain time (like on startup) or with certain applications (like when you use the internet, etc). It would definitely be informative to try the sniffer idea like verdeboy2k suggested, but try and capture some of the outbound packets destined for the 209.178 address rather than the ICMP packets. Give ethereal a try, it's a little more user friendly.
Thanks for the follow up.
I have been gathering more information too.
I rewrote the firewall rules and turned on tcpdump to trace traffic to/from udp 1026 & 1027.
I think I have covered the possibilities now, please see the details below.
I am less worried today because I think I can convince you, below, that there are no packets coming from my machine that could prompt those destination unreachable messages. I am further reassured by your observation that the original packets do not look like Linux packets.
I agree it is definitely weird that the iptables conntrack module is picking up what seem like packets returned to a spoofed ip address. My working hypothesis is that either there is something I do not understand about using the conntrack module or else that the tracking rules it applies are too loose. As a first step I am splitting the firewall log rule between ESTABLISHED and RELATED so that I can see which condition it thinks applies.
I am going to pursue the matter because I think it could be a weakness if the module accepts icmp packets that are not really associated with an established or related connection. The rule I (naively) wrote does not specify a protocol, and, without proof to the contrary, I assume that accepting packets from other protocols could open a vulnerability, and I assume the module maintainers will be concerned too.
I was hoping that someone else had seen this problem before and I could avoid doing this work. If this is really a new observation, I hope you will help me to describe what we learn in a way that will help find this information in the future.
I hope you agree that given that I could not tell if the icmp messages meant that my system was compromised or that the conntrack module had a problem, going to a security forum first was the right thing to do. I guess I do have to learn how to look for bug reports, that is where I guess I should go next, now that I know what part of the code I suspect.
Here are some of the details of what I have found:
First, you are right about the incoming udp packets.
These are clearly spam, and obnoxious spam at that.
I think I have seen some pop-ups in mozilla that warn me
that my windows machine has a vulnerability.
In fact, I have turned off unrequested pop-ups.
So I guess mozilla may listen on one or these ports,
or maybe these particular packets are directed at some IM application,
that I, luckily, do not run.
Here is one line from the tcpdump report:
06:06:27.150595 61.17.xxx.yyy.777 > 66.92.aaa.bbb.1026: [no cksum] udp 1285 (DF) (ttl 106, id 0, len 1313)
"Your system is affected, download the patch from the address below ! "
Anyway, the point is that the firewall is logging the new packets coming in,
and tcpdump is tracing all upd packets to or from 1026 or 1027.
Here is the tcpdump command I used:
tcpdump -i eth0 -p -s0 -w funny-packets2 udp dst port 1026 or udp dst port 1027
OK, so during the time that tcpdump was running, all the upd packets to or from 1026 or 1027 were all incoming, like the above example, there were no outgoing udp packets.
Almost all these udp packets matched the -m state --state NEW rule.
I can match the log and tcpdump entries except at two timestamps. In these cases tcpdump recorded 3 identical packets and the firewall only logged one. I think this is still ok.
Also, in the firewall, I added checks for packets being accepted in or out on udp ports 1026 and 1027 under the -m state --state ESTABLISHED,RELATED rule in the forward, input and output chains. All the udp packets are accounted for, and no udp packets were sent or received with a destination port of 1026 or 1027 through these state rules.
During this time, when there were only incoming packets on upd 1026 and 1027, there were several more instances of the icmp message saying my machine had failed to reach ....
This is the critical step. I got the destination unreachable error messages during a time when I believe I had not sent any such packets.
I think this makes a good case that these icmp messages are about packets with spoofed ip addresses that did not originate on my machine. I think to be super sure I would have to run a packet sniffer on a separate host on the same network, but I think I will go look for information about the iptables conntrack module first.
I hope you agree with my analysis.
I will keep listening to this thread, and I will post any more relevant information.
Please let me know if you think there is anything else I should do.
Thank you for your support.
Well, I am not an expert, for sure, I am searching for that answer now. I got into this by adding a rule to my firewall to see what would happen. I had a rule with the pattern -p icmp -m state --state ESTABLISHED,RELATED . I then added a following rule -m conntrack --ctstate ESTABLISHED,RELATED. I saw I was getting a few matches, so I added a log rule. When I saw what the icmp messages implied, I started to worry.
So far it seems clear that icmp messages can be matched to existing connections, and that udp messages are entered into the connection table even though they are "stateless". It seems that for the icmp destination unreachable message to match the pattern there would have to be an entry in the connection table to match to. However, this module seems fairly complicated, and I may be misusing it out of ignorance. It may be that it is matching some other connection or because the icmp messages are repeating or for some other reason I do not understand. I am trying to go throught the documentation at http://www.netfilter.org/ to learn more. I think the next thing is to try to find a mailing list where I can discuss this with someone more knowledgeable on the operations of this code.
I am skeptical of the idea that someone was port scanning me. It seems like a lot of work to generate phony icmp packets containing information about another packet I was supposed to have sent. Also, sometimes the icmp messages do not come from the same host as the unreachable udp packet was addressed to. That is, sometimes the icmp message is from an apparent gateway of some sort telling me that a host on the gateways network is unreachable (so the source in the icmp packet and the destination in the udp packet are different, but similar). My first concern is to be sure that my machine is not, in fact, trying to send those udp packets to anyone else without my knowledge. If my machine did not send the original udp packet then I do not yet see how sending me an icmp packet about it is going to tell them anything about my ports. But I could easily be missing something, so fill me in if I am, please.
I would agree that the icmp packets are not likely to be forged. Rather the initial udp packet sent from some unknown host to the 209.178.xxx.yyy host is the one more likely to be forged. I would guess that the initial udp packet sent to port 1026 contained Windows Messenger spam and was forged to have jim's IP address. When the remote gateway machine couldn't reach the destination host, it sent an ICMP host unreachable response to the source IP address on the packet. Because that IP was forged, the ICMP packet was sent to Jim's machine instead of the true source (the spammer). While the port 1026 udp packets captured by tcpdump were the same kind of Windows messenger spam, those packets are likely un-related to the initial packet that generated the ICMP packet. In general ICMP scanning is not a very efficient means of information gathering. If one were to go through the effort of crafting specialty packets, there are a lot better ways of doing it, though it still could be possible.
What is curious to me is that the conntrack ctstate rule matched the packet. From my understanding both conntrack and state should handle the ESTABLISHED,RELATED matches identically, in that they should both check the connection state table and should only match if an existing entry is in the table waiting for a reply of some form or another. If you look at the iptables man pages and other documentation (which is sparse at best), the description of the conntrack and state matches are essentially identical with regard to NEW, INVALID, RELATED, and ESTABLISHED states (in fact they are verbatim in the man pages). The only difference I can determine is that with conntrack ctstate, you can access additional information in the state table, but that seems to only give you the added ability to match packets based on DNAT/SNAT information. In this situation, that should not have any impact. The fact that you are not capturing outgoing port 1026 udp packets, would suggest that you shouldn't have any waiting connections in the state table or alternatively that the system has been compromised (which I highly doubt). You can visually verify the state table by looking in /proc/net/ip_conntrack and seeing if you have any waiting udp connections. It might be a good idea to post the question to the iptables mailing list and seeing if anyone has observed this behaviour before. It would appear that the conntrack match is busted, but to be honest I'm not very familiar with the inner workings of the conntrack module, so this could be a feature(?).
Note that the concept of "state" here is differnet from the networking concept of "states" (as in UDP being a "stateless" protocol). In iptables-speak, you can have UDP/ICMP connection states. The iptables term is dependent on whether you have sent a packet and are waiting for a response versus the networking concepts of packet sequence numbering and SYN/ACK/RST flags.
If you're interested on reading up, the best docs I've seen on iptables states and state matching is available through the netfilter site or here: