Hello guys!Thouse of you who are hackers maybe will laugh of my question, anyway I address this email to you. My server was hacked yesterday, I believe from a Norway IP(195.204.1.132 check it yourself). Sure i am interested how they break in,maybe any of you could guess, but i dont think i can give you any of the details of the services or any details of the system. Sure if you want just feel free to ask, i dont list them because i dont know what will be relevant and what not. In few words it is a fedora 4 box, with web/mail/ftp/ssh services. I just SAW a weird thing on my box right after the attack, there were TWO httpd in the memory, one was mine and one was the malicious code inserted on my system. After i see it in memory, I stopped mine ,hoping it will kill the other one. Tried to restart httpd it would not run, complaining could not bind the to the ports 80,443. Sure the "other" httpd was still running. Running the command "killall httpd" didnt help either. I killed it using its PID. Now i just cant understand how is it possible for a process to stay in memory and to pretend to have a name a path, and actually to exist in other part of my filesystem. I didnt find it, cause the other httpd have the same path as mine (/usr/sbin/httpd) , but i think they somehow faked that and it was executed from somewhere on my filesystem. I hope my words make sense.. ANy thougths??

PS: In such days a sysadmin understands all the linux knowleadge of three years is nothing, comparing to a hackers understanding.

Yours, clockbyte.

My server was hacked yesterday
Who has access to this system: is this a private box or public system?
What services are/where running on the box?
Was all software up to date?
Did you run a file integrity checker like Aide, Samhain or even tripwire (if already installed and in use)?
Did you run Chkrootkit/Rootkit Hunter/your package manager (if it has verification option)?
What do the auth/system logs say?
Any new or changed accounts on the box?
Any setuid root or other anomalous files in $TEMP/public writable/user dirs/partitions?


Now i just cant understand how is it possible for a process to stay in memory and to pretend to have a name a path, and actually to exist in other part of my filesystem.
Could be mucking with names like in strcpy(argv[0], "/usr/sbin/httpd");.
Have you found the location of the binary?
If so, what else was there (just post ls -al the dir)?

If you were hacked maybe you binaries have been subverted. Like you killall for instance. Anyway once an attacker gains root access his has full access to do what ever he can with the system, including hiding processes, hiding network connections, hiding files. It is all a question of how good the rootkit is.

And I don't agree that sysadmin are always behind hackers. That is not the general rule, as about 90% of the attackers are noobs, 9% have a clue and 1% are pros. We may be behind that 1% that are the ones that write the tools.

1) Running "killall" sends signal 15 by default. If you can't kill some process run
"killall -9"

2) What did you see and came to the conclusion that you were hacked ?

3) You saw "TWO httpd in the memory"
Just by seeing two httpd processes doesn't mean that one is malicious code.
"lsof" show much information about processess and it can be a valuable tool when you
want to see what some process does.

This is from my box.
The process running as root (PID 227) is the original one that bind the ports etc.
The other processess (Parent PID 227) are running as nobody as you see.
This is normal i believe and i have 4 processess not two.

I don't say that you were or you weren't hacked. I just say that seeing two httpd
processess doesn't necessarily mean that you were hacked.

That IP you specified is just an IRC ip, so if somebody wasn't suppose to be using irc from the box, or you are the only user, chances are you have been compromised.

Running lsof -i and netstat -ptav 'might' reveal some sensitive information, considering they haven't been trojaned. I would highly consider downloading rkhunter and running that, and even chkrootkit. Check ~/.bash_history for any unusual commands, although chances are they simply used unset HISTFILE to disable it.

Once this is all sorted out, I can't stress enough that you install tripwire, or some other IDS.

Good luck.

Thank you all for help! I found out that hackers break in using the mod_perl. They somehow found an user/password for the FTP , cause we are a hosting company, then copied a file in perl, got a SHELL, and installed all the STUFF. Actually the weird thing that i have seen on my server now is somehow clear, they executed the daemon, then deleted it from harddisk!, just that simple, clever guys I would say. I upgraded the mod_perl, it was version 2.0.4, now everything seems to be ok. Now i think I will find time to install a IDS and carefully read the logs, a thing that before i was to lasy to do it. Thank again!

Cliff

They somehow found an user/password for the FTP , cause we are a hosting company,
No, they where able to bruteforce an user/pass combo because you didn't detect it.
You probably also didn't employ a security-conscious FTP daemon and/or didn't configure your FTP server properly.


they executed the daemon, then deleted it from harddisk!, just that simple, clever guys I would say.
Using a tool like Tiger or using lsof would have caught this.


then copied a file in perl, got a SHELL, and installed all the STUFF. (..) I upgraded the mod_perl, it was version 2.0.4, now everything seems to be ok. (..) Now i think I will find time to install a IDS and carefully read the logs, a thing that before i was to lasy to do it.
So they had root access on your box, and (if I'm reading this right) the ONLY thing you did was upgrade mod_perl?!!
Not fsckin good enough. Disconnect the box from all networks, inspect it, backup anything you need except binaries then follow the three r's: repartition and reformat completely, reinstall from scratch. Inspect other boxen for signs of intrusion. Advice anyone passes need to be changed. Properly harden the box.

Anything else is a waste of time and effort.

To unSpawn: They do not got root, only the apache user. I verified with the "find / -newer some_old_file_before_atack" and found their daemons and some music..he-he. They turned my box in kind of file server, copied files on it and allowed their friends all over the world to copy 'em. Thats all I found, i also was thinking of reinstalling my sistem, but it is not needed. I am using VSFTPD (kind of very secure) FTP server,i am not sure is it the best or not, maybe not. So, I dont think the FTP server could be protected against any bruteforcing..?? What ftp do you recomend and how about brute forcing..?

thanks for reply

by installing an IDS you can detect brute forcing. it seems like they were just a script kiddie that turned your box into a server, but like you said before

"In such days a sysadmin understands all the linux knowleadge of three years is nothing, comparing to a hackers understanding."

for all you know, the hacker wanted you to think that they turned you box into a server, specifically for the purpose that you dont reformat your computer. like unSpawn said...

"Disconnect the box from all networks, inspect it, backup anything you need except binaries then follow the three r's: repartition and reformat completely, reinstall from scratch. Inspect other boxen for signs of intrusion. Advice anyone passes need to be changed. Properly harden the box.

Anything else is a waste of time and effort."

They do not got root, only the apache user.
Did you run a vulnerable Apache version: see (Hijacking Apache using Mod_perl)?


I verified with the "find / -newer some_old_file_before_atack"
Do yourself a favour and swap using MAC times for using checksums, that is compare your binaries against "known good" versions of packages from like your distro's mirror (using your distro's package manager if it has checksum verification). MAC times can be manipulated, manipulating checksums is much, much harder AFAIK. The downside of using a package manager to verify checksums is that the scope is narrow: it won't find files outside packages, and that's where installing a file integrity app like Aide, Samhain or even tripwire comes in (also note Rootkit Hunter, L(N)SAT and Tiger posess means of file verification), unfortunately installing a file integrity app only makes sense when the box is still pristine (just after installing the O.S.). "find" can be used as a way to determine if there any files were introduced to the system, but with reservations wrt MAC time manipulation and hiding stuff.


i also was thinking of reinstalling my sistem, but it is not needed.
If you can "prove" the above actually was the case (hijacking) and the rest of the system can be verified as untouched, *then* you could conclude that.


I am using VSFTPD (kind of very secure) FTP server,i am not sure is it the best or not, maybe not. So, I dont think the FTP server could be protected against any bruteforcing..?? What ftp do you recomend and how about brute forcing..?
I used Vsftpd myself, and it's the only FTPd I can recommend if one can't dodge providing FTP.
What you can do wrt bruteforcing depends on what you need to provide (restricted access means easier enforcing), if you run Vsftpd as standalone or from (X)inetd and if you use the PAM stack. Iptables for instance can limit connections on a per-IP+time basis (recent, hashlimit), Xinetd can provide checks like IP/range, instances and adds logging (log_on_failure) you can use with a logreporting tool to take action on similar to those tools used to thwart SSH attacks. The most basic thing is that if you provide anonymous access you should not provide writable access and that FTP-only users should not need an account or shell on the system. PAM provides restrictions like using a different password database, setting allowed login accounts and login failure counts. Of course you can add any other useful module to the stack. Like true_atlantis says IDSes like Snort provide rules to check for attacks, but for that to be effective you will have to add a tool that can block through Iptables like Guardian.


* Wrt to Apache file descriptor crap you're not to blame (that is, only if you *did* upgrade) and the fix should come from the Apache team. Auditing (using env_audit) is a must, and anyone should do that regularly and to test newly installed network-accessable or related software.


HTH.