Linux - SecurityThis forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
First post here, because I really need help! I think I got massively hacked!
I noticed some network instability last week with the internet going down, so I kept my eye things. Everything seemed to be okay, but now today I notice for the last couple of days thousands of packets going OUT from some of my LAN computers to pings, weird ports like "nvp-ii" and all kinds of other weird anomalies. I was away for the weekend, and just logged into the firewall today to check. I've never seen anything like this!
for example:
Code:
From 10 - 1 packet to tcp(57111)
From 192C - 1 packet to tcp(58226)
From 192.1.254.113.98 - 1 packet to tcp(58226)
From 192.168.0.1 - 444 packets to udp (xxxx)
From 192.168.0.112 - 15269 packets to ggp(0)udp(53,123)
tcp(0,21,22,80,110,443,36267,37372,37373,39498, etc.)
tcpes(0)icmp(0)
From 192.168.0.113 - 3197 packets to nvp-ii(50814)tcp109(80)il(80)t2306(51379)ip(50814)
tcdf(80)tcp(80,1690,etc. udp(53)td(51379)tcp66(50814)t(80)
From 192.168.0.116 - 3824 packets to udp(53)tcp(8UT,80,222,443,36743,39591,43363,50086,54781)icmp(0)
From 192.168.0.120 - 3183 packets to udp(53,80)tcp(21,25,80,110,443,46287,48326)
From 192.168.0.121 - 7666
192.1.254.113.98 is not on my lan, the subnets are 192.168.0. and what is that about packets from "10" and "192C"?
or else it's a weird interface listed, like
Code:
Logged 1 packet on interface ekIN
From 192.168.0.116 - 1 packet to tcp(80)
or again with weird IPs or ones that are not supposed to be on my LAN:
Code:
Logged 2 packets on interface et
From 67.159.31.14 - 2 packets to tcp(43147,53039)
Logged 37589 packets on interface eth0
From 192.168.0.10ckIN - 1 packet to tcp(45030)
So, to back up for a moment, you're running RHEL / CentOS / Fedora, and you've posted a logwatch analysis? There is not enough information here to know whether you've been cracked, but it does seem that there are a lot of malformed entries in your logs, or logwatch is behaving badly.
How about providing: distro, version, logwatch version... Any other sysadmins on the box who may have changed something lately?
So, to back up for a moment, you're running RHEL / CentOS / Fedora, and you've posted a logwatch analysis? There is not enough information here to know whether you've been cracked, but it does seem that there are a lot of malformed entries in your logs, or logwatch is behaving badly.
How about providing: distro, version, logwatch version... Any other sysadmins on the box who may have changed something lately?
2 Fedora 8 boxes, 2 Arch linux boxes, 1 Debian testing box. The firewall is IPcop, current version, and the logwatch on there is whatever version is current on ipcop. No other sysadmins, just me.
You think maybe just the logs are messed up? that one weird lan IP especially really has me worried. 192.1.254.113.98 -- what on earth even is that?
Go to the source an look at the actual logs, not another programs (possibly incorrect) interpretation.
IPv4 addresses as you know are 4 numbers, separated by dots. The fifth number may be a port number (a common practice). What do the logs show?
The 192C most likely means a Class C 192 network, so in your case, all of 192.168.0.0/24.
You're saying things like "I've never seen anything like this". Problem is, what don't know exactly what you mean by "this". Please be more specific.
Thanks Mr. C. I've been checking most of the logs on the other machines (2 are down), but I don't see anything too weird. Just on the firewall.
By "this" I meant IPs in the log summary like 192.1.254.113.98, 10, and the 192C, the strange interface designations like "Logged 1 packet on interface ekIN," or the tens of thousands of packets going out from these computers, some of them pinging out every second all night long. I've never seen behavior like that, either in the logs, or my network computers pinging china and the internet all night.
One user just told me, "I was trying to download something called a torrent file, but I couldn't get it to work." I thought maybe they got hacked through some fake torrent and now the whole network got compromised.
Last edited by userlander; 07-21-2008 at 06:10 PM.
I agree with Mr. C... Check the original firewall logs.
I would also add that unless the network card has failed in a odd way, there won't be traffic coming out of your computer unless it is from a process. Because of that, post the ps, netstat, and lsof output from the computer you suspect is compromised, and hope there isn't a rootkit installed that is hiding evidence.
Perhaps first, you should tell us whether this is your home network, or your employers? If it's your employers, do you have an incident response policy? What services were internet facing? Was it routinely patched?
I agree with Mr. C... Check the original firewall logs.
I would also add that unless the network card has failed in a odd way, there won't be traffic coming out of your computer unless it is from a process. Because of that, post the ps, netstat, and lsof output from the computer you suspect is compromised, and hope there isn't a rootkit installed that is hiding evidence.
Perhaps first, you should tell us whether this is your home network, or your employers? If it's your employers, do you have an incident response policy? What services were internet facing? Was it routinely patched?
Hi OlRoy. To your questions, this is a home network. I don't think my employer has ever even heard of linux. :-p I have a web server on the internet, and that's it. I have an FTP server on the web server, but that is only open to the Lan and 2 external IPs known to me. SSH is only open to the Lan and 1 of those IPs. NTPd runs on the ipcop box, and then serves the Lan with the time.
I checked the ipcop logs and did not see anything unsual. I grepped for some of the weird IPs and didn't find anything. I then ran chkrootkit and rkhunter on all the boxes (except the one that's down) and all those returned no findings, either.
The original computer that I suspected is offline now (disconnected) and I can't check it until tomorrow or later in the week. That's the one that was pinging the internet. I haven't noticed the other ones were doing that, but there are some pings from the router to two of my lan computers, which maybe seems weird as I definitely didn't do that. lsof is outputting a lot of data, I can't really see anything too suspicious, but there's a lot of entries to sort through. Are there preferred options for narrowing down the results?
netstats looks normal. here's a typical one:
Code:
11:05 PM:~ $ netstat -al
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 *:52168 *:* LISTEN
tcp 0 0 *:sunrpc *:* LISTEN
tcp 0 0 *:33564 *:* LISTEN
tcp 0 0 192.168.0.116:53756 desk:49444 TIME_WAIT
tcp 0 0 192.168.0.116:49880 ipcop:222 ESTABLISHED
tcp 0 0 192.168.0.116:686 server:nfs ESTABLISHED
tcp 0 0 192.168.0.116:936 desk:nfs ESTABLISHED
udp 0 0 *:57237 *:*
udp 0 0 192.168.0.116:45879 ipcop:ntp ESTABLISHED
udp 0 0 *:735 *:*
udp 0 0 *:sunrpc *:*
That's all I can think of for now. Maybe the logs did just get corrupted. But why would that one computer be pinging the internet all night when no one was on it? That would seem to mean it was some bot that got installed on there scanning for computers to hack, no?
Stop randomly speculating! The data is there in your logs; when you look at it, you can confirm or reject any theories.
The messages log itself has over 260,000 lines in it -- what exactly should I look for? Can I narrow it down at all? Today the log summary is even more malformed than ever. Is there any way to test the integrity of logwatch?
Try to reconcile some of the weird logwatch results you're seeing with what iptables is logging. If you find matches, post some of the log entries here.
Try to reconcile some of the weird logwatch results you're seeing with what iptables is logging. If you find matches, post some of the log entries here.
Like I said, I searched for some of the weirder ones, with commands like: cat /var/log/messages |grep 192.1.254.113.98, or |grep 192C, but I didn't see anything. I searched for the IP that was pinging along with grep -i icmp, but that's already back a few days in the logs and I haven't found out which messages.n.gz it's in yet.
Today in the logwatch summary I have interface e, eoe, et0, eteth0, eth, eth00, and eth0IN. Here's what I found searching for some of those (some I couldn't search for, like just plain e or eth, for example):
I notice now that all those are involving my web server (server, 192.168.0.113). The destination IPs are variously in Russia, France, Germany, and the US. I don't see any real pattern. I looked through the auth.logs, apache logs, and message and syslog on the webserver and didn't see anything related to those IPs.
I'm stuck, what should I do next?
----> Oh wait. In the older apache logs, a couple of those IPs showed up so far, with harmless access. They were all accessing the same graphic. It must be related to that somehow, like maybe it's too big or has corrupted data in it or something. Still searching for the other ones...
Yes, they're all there, and they're all accessing the same graphic. That must be it, how weird.
Last edited by userlander; 07-22-2008 at 10:57 AM.
Your touching the data - don't walk away from it until you have satisfactory answers that add up and make sense.
I would investigate the reason for the log formatting anomalies. Use:
grep eteth0 | od -bc
to determine exactly which characters are in the log line. This could be intentionally placed non-printing characters, to cover tracks. If this looks fine, then I would start looking towards other causes. Be sure you actually don't have those interfaces defined (ifconfig -a). I would also perform a full RAM and hardware test, as well as running a full fsck on all the file systems. If all those pass, at least you can trust the hardware and the file system integrity.
The image being accessed - is that on your server, or on another server? Your system may have been turned into a bot, and the image is referenced in spam email, or a rogue website. Are these questionable requests incoming or outgoing requests?
Your touching the data - don't walk away from it until you have satisfactory answers that add up and make sense.
I would investigate the reason for the log formatting anomalies. Use:
grep eteth0 | od -bc
to determine exactly which characters are in the log line. This could be intentionally placed non-printing characters, to cover tracks. If this looks fine, then I would start looking towards other causes. Be sure you actually don't have those interfaces defined (ifconfig -a). I would also perform a full RAM and hardware test, as well as running a full fsck on all the file systems. If all those pass, at least you can trust the hardware and the file system integrity.
The image being accessed - is that on your server, or on another server? Your system may have been turned into a bot, and the image is referenced in spam email, or a rogue website. Are these questionable requests incoming or outgoing requests?
Thanks for all the help. When I tried grep eteth0 |od -bc, it says od command not found. It must not be on ipcop.
The image is on my server. The requests are incoming to access the graphic, and so that seems okay, except for how they're showing up malformed in the log. I can't believe those would be malicious attempts though, because they're all coming from different parts of the world and are all accessing that same image. It seems like that would be way too much of a coincidence. I have to believe it has something to do with the graphic.
The internet just "locked up" again, though. I was able to ssh in to the router just long enough to run top, ps ax, and iptraf, found one IP connected through port 33599 (or attempting to connect?), and then the connection locked up. I didn't see anything using up CPU in top, so I figured it must be the network causing the lag. I had to make a new connection through the server to connect, which got me in just long enough to reboot. If I wait too long, it just completely locks up, and I can't access it or the internet at all.
Maybe the network card on the internet side is bad?
But you *do* have od on other *nix systems. Transfer the log file! This is what I mean by "Your touching the data - don't walk away". You've walked away again, without an answer.
You have ot make up your mind. Either you want to diagnose the possible problem, or you don't. I hear "HELP" in your Subject, but hear "meehhh, I'll speculate instead" in your responses.
I don't know what you mean about coincidence regarding the graphic; it sounds like you're saying malicious attempts don't come from random IP addresses. This theory of yours is wrong. In fact, that's exactly the pattern seen when a system has been compromised in some way.
I won't speculate any longer here. You have the data, examine it or don't. Is your call.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
I thought maybe they got hacked through some fake torrent and now the whole network got compromised.
