Help analyzing hackers transcripts
I am a newbie Linux sysadmin (but seasoned Windows administrator), and I have five boxes running in a test environment. I thought that until we went live with our production servers, we were safe, and we could let our guard down. But, after I got a Nagios [monitoring service] alert that my SQL server was running a large number of processes, I knew something was wrong.
I logged on and ran a ps -e command. It came back that there were a large numbers of the ssh-scan command running under the pts/2 terminal. But who and users showed that I was the only person on. I did a killall ssh-scan to eliminate the threat, cleaned up the rest of the processes in the pts/2 command ps -e | grep pts/2, changed the root password to a good, strong password, and thought I was done. Except I wasn't. Two days later, we went through the same routine. Then after that as well. Someone told me that bash logs every command, so I ran locate .bash_history. I came up with our users, one in the root directory, and one in the /var/lib/mysql folder. what? We don't use MySQL, we only use PostgreSQL. I found my clue! Unfortunetly, that's where I ran out of knowledge. I tried googling a few of the commands, but I wasn't able to figure out the breadth of the attack. I am posting the entire transcript here, maybe someone can tell me where I should be loooking to find more files that he may have dropped on our hard drive, or backdoors that he may have created. At the very least, it can be indexed by Google, and save someone else a bit of trouble. -Simon Code:
cd / |
A bit more!
After I posted the question, I looked at it again, and saw the logs seemed to gravitate around the /var/tmp folder quite a bit, so I wanted to start poking around there. I ran a ls, but it came up empty, so I tried again, this time with a ls -a. Now I got something, . . ... I knew from a previous poster that when he was hacked, he had found files installed in a directory called ". ", or a dot followed by a space, and I tried that. It worked! I dropped into /var/tmp/. ! Another ls -a, and I saw another hidden folder, .S008. Below this, I found my portscanner! I have posted the README for indexing.
I also see that all the files were owned by the Postgres user. I guess we forgot to change the password when we installed it. Oops. btw, the I uninstalled mySQL right away. Hopefully, that puts an end to the problems right there. Quote:
Quote:
|
Quote:
|
Since you already got one *good* reply I'll try and expand a bit more.
Quote:
Quote:
Quote:
Quote:
Quote:
Basic rule for "victims": first read, then make a plan, then execute. Not the other way around ;-p Getting acquainted with the Intruder Detection Checklist (CERT): http://www.cert.org/tech_tips/intrud...checklist.html and Steps for Recovering from a UNIX or NT System Compromise (CERT): http://www.cert.org/tech_tips/root_compromise.html should be good always. And when it's time to rebuild have a look at the LQ FAQ: Security references: http://www.linuxquestions.org/questi...threadid=45261 or ask for tips in the Linux Security forum about system hardening. HTH |
Quote:
@SimonHova: Take his presence as a "courtesy" warning and wake up call. The next guy may not be so clumsy. |
Quote:
* Forgot to say that if one box in a set gets compromised, check the others as well. |
The "atacker" is a newbie script kidie from romania that found some tutorials and programs on DC++ about hacking and stumbled on your server. With absolutely no security it was like a playground for him. He will probably not come back, but just in case monitor after an romanian ip (the censored from "wget {censored}/mech-linux.tgz " might be his own IP adress).
Why I i say he is a newbie from romania: because he didn't erase his tracks and the language from the README you posted is in romanian (i fall off my chair from laughter when i read it because romanian isn't such a popular language :))). I knew a couple of these kind of hackers and after they "hacked" a server, they never returned :D They just did it for fun, to brag to his friends and to have a IRC bot. Also, look in the logs from where the ssh connections came from and monitor, in case he cames back for more fun. To be safe, install a newer version of your distro with all the latest security patches applied. |
Moved: This thread is more suitable in <SECURITY> and has been moved accordingly to help your thread/question get the exposure it deserves.
|
I'd look at the timestamps for files in the " " directory in /var/tmp and try to correlate it with the system logs that happened around that time. Perhaps you will find a bunch of failed SSH password attempts and then one successful one, or that a service crashed and shortly after a new user created, etc. If the attacker didn't remove his tracks from .bash_history, there is a chance he didn't modify other logs.
|
After reading this thread I think one important piece of advice has been missed and that is, that a compromised server should never be "cleaned up" and used again with the same operating system image. Even novice Romanian script kiddies have prepackaged backdoors and as such there is always a chance that your system has been rootkit'd. Therefore my recommendation is always to image the system and then completely re-install it. Of course making sure that all of the latest patches have been installed before making it live and adding extra hardening option such as selinux and grsec.
You can then investigate the break in on the imaged drive which is more likely to be accurate since rootkits can manipulate the integrity of the system, such as by hiding files and so forth. |
In general I agree with jamesapnic, but there are usually exceptions to the rule. What if the attacker only had user privileges? If management can't tolerate the downtime? If you have complete bash_history list of what the attacker did and the knowledge to fix it? What if there isn't any evidence of a rootkit? What if you're filtering outbound traffic and the attacker wasn't even able to put his tools on your system? Or if the service is chrooted and there isn't any evidence of him escaping it?
I think there are a number of things to consider. Each incident is different so the response should be adjusted accordingly. |
Quote:
For instance most of the times the OP will not post all the required nfo. While it's tempting to post advice "nanananah, I wedged in my reply first!!!" the *best* way to start off would be to *ask questions* to clear things up. Once you completed such an assessment you have a picture of things and it would be more effective to base advice on that. If anyone is interested in handling incidents "the right way" (at least in my opinion), please check back this forums previous incident threads (esp. older threads), read the required stuff on the 'net and at LQ (and for instance recaps like http://www.linuxquestions.org/questi...l?#post2291514) and just go for it. |
Judging from where you found the initial traces, they probably used a weakness in one of your web applications to gain access, possibly in conjunction with a SQL injection attack or leveraging flaws in the database configuration. I'd say that because the home directory was /var/lib/mysql, even though you use PostgreSQL. They probably just ran a generic PHP/SQL attack and their script assumes that it's MySQL. Of course the other possibility is that you do have MySQL running as part of one of the applications you installed, so even though you're only consciously running PostgreSQL for your web apps, you actually have MySQL running too (and since you didn't realize it was running, you didn't take precautions to lock it down).
In any case, the vast majority of external break-ins these days come from poor web application security, so go over any web/db services running on that machine with a fine-toothed comb. |
Quote:
Quote:
|
UnSpawn to the rescue again.... I totally agree with him. There is alot of time spent into incident response and handling. It is not something that can be learned overnight. It takes years and years of knowledge. Understanding how the system operates at the system level and/or kernel level. While there are always hackers out there i would recommend that the OP looks into extra security for the servers. There are hunderds of options that he can do to increase the security of the system.
kernel level: Grsecurity SELinux RBAC LIDS application level: apache mod security hardened PHP chroot processes General security: disable root ssh in /etc/ssh/sshd_config remote syslog server that the systems log to. Daily emails from the servers Enforce strong passwords Most of the stuff above will keep out the vast majority of script kiddies. If there is someone out there that wants in your system and he is good enough then he will get in. BUT most of the people that have the knowledge to get into a hardened server are not going to bother with it. They have more important things to do then hack _your_ system. |
All times are GMT -5. The time now is 10:48 PM. |