Server hacked
 

Hi guys,

Recently my server was hacked. I have lots of questions for you Linux experts.

1.And I was wondering if there is any turnkey solution to check out the vulnerabilities of the server. E.g. the software should be able to simulate a hacker and try to hack into my server then notify me of the vulnerability.

The information on Linux security is just too overwhelming so I was thinking if there is any easier solution for this. If not then someone should try compile one as I believe it would benefit Linux newbies like me.

2.As for my server, someone hacked into it by uploading phpmyshell program.
But how can he gain access to other accounts from there??

3. He was dumb enough to leave traces in my /var/log/wtmp
I got his IP address and the time he logs in.
I went to FTP section and downloaded the raw FTP log.
I nabbed that fella!

212.174.89.155 - - [25/Jun/2004:06:51:20 -0400] "GET / HTTP/1.1" 200 660 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; 118K501TUR)"

Went to http://www.ip2location.com/free.asp to check out the IP:212.174.89.155

"212.174.89.155 TR TURKEY"
Got him!

Then I used IP tables to block the whole class C IP.OK I am mean. lol
iptables -I INPUT -s 212.174.89.0/24 -j DROP

Now what should I do with it?

4. If he wasn't smart enough to cover his traces I assume there must be a way for me to find out which files he modified, right? OK how????

Thanks in advance for your help! I will update you guys regarding the situation.

now that you have done your research, it is always recommended to format and reinstall the OS.

Don't forget to report him to his ISP.

The sad part is I am running web hosting service so reinstalling would mean my clients have to upload their site again.

I will try to avoid this if possible that is why I asked question number 4.
4. If he wasn't smart enough to cover his traces I assume there must be a way for me to find out which files he modified, right? OK how????
I wanted to just reverse the damage done to the system.

Take a look at the /var/log/security log and messages log. Might give you some clues. If you were not running Tripwire previous to the breakin, then there is no telling what files the hacker changed or created.

I would be real surprised if that IP you got in your research turns out to be the culprit. More than likely, that is just a IP of a system the hacker compromised and used to run the exploit. And that IP is in Turkey.....good luck...he is long gone.

The previous suggestion of format/reinstall is right. After reinstall, but before putting the box back on the network, install and run Tripwire. It will create checksums for all your executables so if anything changes, you will know.

Later...

If he has got a router in his home and a proxy server in Iran, it will not do much good too report him to his ISP, as they would probably tell you to speak to their chairman or police and most crackers above the age of knowledge dont like to get arrested.

He is from Turkey. When he defaced the site it was showing some 'Turkish Pride' message with turkish flags.

There may be other trojans/backdoors lurking there which you might detect pretty late. And till the time you weed out all the compromised programs, you will be giving them time to carry on their activities.

If you are really keen, take a dump of the disk for forensics. Repeat, format and reinstall the OS.

Although there may be some downtime, you can be rest assured that your new OS is free of all trojans that may be compromising your clients' data.

BTW are your clients aware of this break-in?

They are aware of it. So it is impossible to reverse all damage?
I really don't want to reinstall as I have done many mods and configurations to the server.

May be it is possible to reverse all the damage.... but can you be sure there isn't just one more backdoor that has gone undetected?

You can always take a backup of your config files.

Anyway ever since I used IP tables to block the whole of Turkey from accessing my site I never had anymore break in from them.
They used to break in like everyday

jut a thought ... perhaps you can setup a honey pot just to monitor the TR user's intentions, etc.

Youmean honey pot as in real honeypot or somekinda linux software?
sorry Im new to linux but Im learning real fast.

How did you block the entire of Turkey from accessing your Web Servers / Proxy Servers?

If you do not want to reinstall. Change your Root password and User passwords. Make sure passwords are safe (plenty of numbers and no dictionary words). Also consider setting up a firewall.

Which firewall do you recommend?

I would suggest buying a router, and following its manual to create an effective firewall. I saw a wireless one in PC World being sold for £47, last week.
It is possible to use a Linux box to act as a router, but that is well beyond my skills at the moment.

What about software firewall?

If your machine has indeed been compromised (esp. multiple times) then re-installing from trusted media is the only way to be sure that a cracker hasn't planted backdoors and rootkits on the system. Using a firewall and/or changing the root password is not enough. In fact by continuing to run a potentially compromised system, you are putting your clients and other systems around you at increased risk as well.

You can backup any human readable files or things you can verify (for example by md5sum), but all other files including binaries and un-verifiable client files should not be retained. Taking the time now to address the compromise properly and to put some forethought into a real security strategy will save you much more time and headaches in the long run. Trying to salvage a cracked box that has clearly shown to be an easy target and may have hidden daemons, sniffers, kernel modules installed on it is really a poor choice.

After you reinstall, might be a good idea to create a master backup of your /usr / and /etc dirs, and keep it on cd to deal with these unfortunate occurences. Its a lot easier to restore a couple of images from cd than reinstall.

What if the /usr and /etc foders are "infected"?

For a firewall i would suggest Ipcop or Smoothwall, set them up on a dedicated machine, you can then use the firewall rules and IDS (intrusion detection system) to block unwanted intruders. They are both really easy to configure and usually you'll be up and running within the hour.

I was recommended APF for firewall.

Anyway I might shell out $90 for this service.
http://www.rfxnetworks.com/linux_appsec_secbundle.php

I have the feeling that this will tighten my security to the MAX.
Do you think I could have done it myself? It looks like a lot of work. Definitely not something for newbie like me.

My recommendation would be to re-install as well. Although you may be able to "reverse" the damage, you would never know if anything else was left behind.

I guess one way you could find what changes were made is to compare the time and date stamps in your log files with the range of files and folders amended within the same timeline. I do understand that you run an ISP service and maybe you could move them (your clients) over to a secondary server while you rebuild your existing server and maybe you ought to seriously consider hardening your server before ever exposing it to the internet. There are plenty of good firewalls. It all depends on what you are willing to spend. A good "free" firewall is fwbuilder you can find at sourceforge.com. However your firewall should be separate from your server e.g. DMZ. I also suggest reading up on UnSpawn's security reference guide

What distribution are you using by the way?

I'm not sure if we are allowed to post services here. if this is not allowed, please remove this post.

cpanelskindepot, if you are in dire need of help I know someone who works at Cisco as a security expert who may be willing to help you at a cost of course. Let me know if you are interested.

Obie, hiring someone who works at Cisco sounds like a lot of money.
I think I will invest $90 in the service I mentioned above.
Anyway if the posting of service related URL is prohibited, moderators please tell me and I will remove it.
I am in no way related to the owner of that website.

cpanelskindepot,

I can still ask him. I'll post a reply here if he agrees.

In any case, like the numerous posts here my suggestion is to re-build your server. Hopefully you have a backup of your files prior to being "hacked". Also you may want to consider rebuilding with a hardened "distribution" such as OpenBSD or FreeBSD. If you are using Red Hat (since I'm more exposed to it than other distributions), it's pretty easy and straightforward to harden your server.

So people can advertise their services on this site? :)

I know a Linux/Windows/Novell/Unix/Cisco guy who can recover lost data, set up a secure network, repair your Grandmothers Desktop and install what ever you want him to install on your bosses home box. Get your PA to become a CCNA, fly you to the moon for tele sponsorship deal, and Bootleg you a Win98 install for a beer, as I am Gill Gates alta ego Pirate black Beard and all my pirated software is protected by the data protection act.
Terms and conditions apply.
Mods please read this thread before you think about closing it. There is nothing offencive or illegal there. Anyway Mods you Rock:)

I think I will go with a rollback. I have a server backup from 14th of this month. And I will isntall tripwire, and harden my kernel (anyone know of any good tutorial for hardening redhat kernel?). Then hopefully it will not get hacked again.

Mods please read this thread before you think about closing it
Thanks, we have been :)

There is nothing offencive or illegal there
No, but advertising is against the site rules, so please keep the thread on topic. If you wish to discuss/offer comercial services and whatnot, do so off of the forums or see our advertising page. Thank you.

Yeah it gets a little out of control now.

And I will isntall tripwire, and harden my kernel (anyone know of any good tutorial for hardening redhat kernel?)
See the section of the security references thread on kernel hardening. You might also want to take a look at grsecurity. If you are going to use an RPM based distro, then you definitely should use an automatic update tool like up2date or YUM. Both will automagically keep your box updated with the most recent security patches. Remember to turn off un-necessary services, use encryption when feasible (ssh vs. telnet), and use a decent firewall. That should eliminate most of the garbage 'sploits that can be used to compromise poorly maintained systems.

Thanks for that caveman!

Anyone who can think of stuff that I should do in addition to what caveman mentioned please tell me. I cant afford to let some Turkish script kiddies compromise my security and defaced my site with some Animated Turkish flag GIF and bad color combination and backgrounds. Hackers suck at design. Why don't we see more hackers deface sites and replacing em with nice flash templates? I will appreciate that!

I did not know that advertising was not allowed in forums, will take that into account. Forgive my attempt at humour.
I think Tripwire is good if your experienced in Linux. But a router is a lot easier to set up and offers the same security. Plus if a cracker somehow manages to bring down the network. The router will ussually automatically detect most of your network settings, upon reinstall.

Just to follow up. I suggest using tripwire aswell.

If you make master copies right after a reinstallation of operating system and configuration of your services, why would you think they'd be infected. Unless you ran your box without any security at all, they should be ok.

Get a hardware firewall/router to put in between your box and the wan.

And why is a hardware firewall better than software?
I was told the software firewall, APF, is more than enough.

A router will make an efficient firewall for you, but as its hardware it will cost you £ rather than time. You can get the same results with software firewall, like TripWire, but unless you know what you are doing, you are likely to trip yourself up rather than the hacker. Hardware will cost your £ and software will cost you time, your choice.

A hardware firewall is better simply because the code is running usually in rom, in an embedded system that cant be changed. A software firewall is just that software, that can be changed, altered, and otherwise crash. By the way cpanelskindepot are you going to reinstall? I tired to take the easy route once and left a compromised system run, and just shored up my firewall, Well it didnt matter, most firewalls are stateful, and will let out any packet that is sent from within the trusted part of the network. Who's to say that they dont have some server/client connection being established via a trojan. If they did its gonna cruise right on through your firewall, even if you block the ip addresses. Yeah man I would reinstall , install some kinda IDS, namely snort, install tripwire, and setup a honeypot that looks alot like the old system they hacked. I bet money they are bouncing from proxy to proxy so bad that you will never find them. Heck you may even want to try a deadzone. just use a network bridge and change protocols on them you could go from TCP/IP then to IPX/SPX and then back to TCP/IP. That would make it go slower though if its a webserver but they couldnt even get to it then.

I managed to get a rollback from my backup on the 14th.....thats like 10 days before I got hacked.
WHen I did, I found some traces of the hacker's activity on the server.
So that guy had been trying for sometime before he defaced my site.
BUT the script I believe he used was not present.

I bet at this point some of you wanted to see the defaced site.
WARNING : THis is not going to be pretty.

http://www.alphaillusion.com/test/Mu...%20_______.htm

If you read the lines below,it said:If you want hack this server, please go to http://www.cpanelskindepot.com/~demo/.admin.php
Dear CpanelSkindepot del it quickly

I didn't see the .admin.php in the /home/demo/ directory so I suppose the have not figured out the admin.php by that time. I am just banking on this fact.
Anyway why did they lead me to the script they used to hack my site and were kind enough to advice me to delete it instead of saying "Your server is lame, we hacked in because your security sucks!!!!"
The only thing I can think of is they actually employed another way to hack in but try to lead me to the wrong script.
I might be thinking too much about all these conspiracy theories though.

I doubt those Turkish script kiddies bothered to find proxy. It is not as if theres Bill Gates bank account password in there. Yes I sell software on my site but my licensing server is somewhere else so they get nothing out of it.

By the way what is honeypot? what is deadzone?
I have no idea what these are as I am really a Linux newbie but they sound pretty good.
I will employ any tactics to get Bozos out of my server.

IMNSHO this thread has been going on too long, with too much advice. Not that I want to hold back any, but you should focus on system restoration, making sure the system is in working order and under your control, and hardening. Honeypots and such are fine, but won't do you no good as they will not enhance security.

Maybe start with these:
Did you update all software?
What services do you run?
What measures did you take to log access?
What measures did you take to shield access?

1. Yes all updated
3. I rely on AWstats for log analysis.
4. So far I only used IPtables to block the whole of Turkey from accessing my site.

And I only need an advice on ONE THING now.

How can a hacker can compromise security and access other accounts in /home/ directory if he was ONLY given the permission to upload a phpMyshell file?
I hope by understanding this I will be able to learn from this mistake.

My server was restored from backup so I guess I am going to be OK.
And I hope others will learn from this too!

What is a honeypot, chechout this site http://www.tracking-hackers.com/papers/honeypots.html

That stuff is hurting my brain so bad.... :(

As suggested earlier. Go get yourself an decent router. Set it up and it will do most of the work that Tripwire and Honeypots can do for you. Backup your critical data, reinstall and study Tripwire and other security software, safe in the hands of the router, that will slap 90% of crackers.

2. What services do you run?
You don't know?


What measures did you take to log access?
3. I rely on AWstats for log analysis.
No, that's webstats only AFAIK. You need to watch syslog and (Chkrootkit, Rootkit Hunter, Tiger, server, IDS, filesystem integrity) application logs for anomalies, set yourself up with a remote email account you check regularly and have something like Logwatch report to you.


What measures did you take to shield access?
4. So far I only used IPtables to block the whole of Turkey from accessing my site.
Start by running everything (except high volume services like HTTP(S)) through LOG target rules. Logging rules major. It also helps you debug rules. On a public webserver the only "established, related" outbound connections are return traffic for the services you run. Initialising (that's SYN for TCP) outbound are DNS queries (TCP and UDP) for resolving and SMTP for sending email, so they need "established,related" inbound. Note some SMTP hosts require you to allow them access to "ident" service (or at least not DROPping them). The only initialising inbound you get are ident for SMTP, (SSH for your remote management caps if necessary (don't log in as root)), and the services you run (hopefully only HTTP(S)). If you're behind a shared firewall your colo ppl might be able to assist by only allowing traffic in and out for the services you need to run.


And I only need an advice on ONE THING now.How can a hacker can compromise security and access other accounts in /home/ directory if he was ONLY given the permission to upload a phpMyshell file? I hope by understanding this I will be able to learn from this mistake.
Upload and run more likely, eh? Remote shell access for unprivileged users in general is BAD NEWS. By allowing PHP to be a GUI for shell commands it's only making it easier. PHP's safe mode would have killed PHP.*shell exec's. Don't trust users to upload, make, modify and run binaries you haven't tested yourself. Don't allow users write access to public (tmp) dirs to create setuid (root) binaries. Don't allow users to execute anything outside the $PATH. Don't trust users (period)


Please read the LQ FAQ: Security references and forget about Honeypots. It ain't helping you secure your box, no matter who mentions it for whatever compelling reason (with all due respect etc, etc).

I wouldn't use a honeypot anyway, i only put a reference to it because someone wanted to know what it was. You would need to have a good understanding of logging and security to run one.

As suggested before, a router is definetly the way to go. Their extremely cheap nowadays, and offer a host of features that anyone could possible want.

Exacly what I said. Glad someone agrees with me.