Hello all,
We have recently made a server available by giving it a global ip. I had set a firewall (allowing http, ssh etc).
It was working fine but before 2 days I found that the root passward was changed. I have set a new password ,but now the system is not starting up normally.While starting most of the services it gives the following error:
'/bin/unicode_start some page number ... segmentation fault'

Some of the commands like vi are also altered(that are in /bin/),and hence not working. The error given while running the vi command is again: segmentation fault.
Is there any way to restore the system to a previos safe state?
Awaiting you replies.

Mahesh.

you can run a root kit checker, e.g. chkrootkit, but as you've seen something as significant as a root password change, can you EVER be confident the system is OK again? If you haven't got backups to restore to you should rebuild the server from scratch.

This is not the known advocated way to respond to a (perceived) breach of security. If you do not have the time, ability or inclination to respond properly, please limit yourself to asking the OP to wait for an incident response handler to pick up the case and maybe posting the CERT Intruder Detection Checklist link, TIA.



Short answer: no. Long answer: No and nor should you aim to re-install the OS from scratch unless you have investigated and know how this root compromise occurred.


Root is not simply an account for human use. Blithely changing the root password is like sticking your head in the sand. On noticing the password was changed you should immediately have asked users whom you share root access with if they changed it or else have isolated the machine by minimally raising the firewall, listing and saving open files, user, process and network details, copying off all logs and accounting data to a different trusted machine and started the investigation or ask service personnel or an experienced systems administrator to take on the case.


If you are the only person in charge of this possibly compromised machine I suggest the following order of things:
0. Please do not make any more changes to the OS, alert (management and) legitimate users the server will temporarily be unavailable. Raise the firewall to only accept traffic to and from your management IP or range. List process details running (adjust "/path/to/data.txt" to be in shared memory like /dev/shm or piped over SSH or netcat to a physically different, trusted machine).
- If the server can not be off-lined temporarily due to the purpose of the server then this is the time to activate your spare or failover server.
- If you don't have any then, if making a bit-by-bit copy is not feasible, at least salvage contents of the /etc, /var/log, web server, /tmp and non-root user home directories and save together with the listings to a physically different, trusted machine.
- If none of this works due to segfaulting tools then if the machine is in colo ask service personnel to isolate it from the network completely until you or they have managed to salvage necessary files.
- Note that if this is a machine in colo you ask a restore for or if you will be re-installing the OS from scratch you may end up exposing the same vulnerability or weakness that lead to the compromise: ensure you expose the machine only after extensive hardening.


1. Next please read the Intruder Detection Checklist (CERT): http://web.archive.org/web/200801092...checklist.html and ask questions before performing tasks. In the meanwhile we ask you post, in as much detail as possible:

- location and type (home, shared, vserver,cloud, etc) and purpose of the machine,
- used OS, distribution, release,
- a list of running services and names and versions of applications installed, including those running on top of the web server,
- date password being changed,
- date anomalies were encountered before this (if any),
- any system, daemon and firewall logs, audit and auth data and user shell history files you might have already looked at,
- any (setuid root?) or other oddly named or placed files you might have noticed,
- anything else you think might be good to know about this situation.

Please respond timely, as verbose as possible and please stay with the thread until closure, TIA

Thanks Unspawn, we can always count on you to be... you.

TIA for staying on topic.

Thank you very much for the concern you have shown, Spawn.
I didn't go for re-installation. I found that several binaries in the /bin/, /etc/rc.d/init.d directories were corrupted, and attributes of some files were changed( files were made immutable) and the log files were deleted.

• We are running content management service on the machine.
• Its on RHEL5.
• Its the first time we faced such an attack.

I think I have recovered most of the things. I copied the binary files from a healthy machine. But I'm not confident yet. I know I will have to be alert.
I will be in touch with you. Yet, the machine is not made available publicly.
Thank you again.

That's about the only good thing to come out of this so far.


Setting the immutable bit is a dead giveaway and there's quite a few kits dropping files in /etc/rc.d/init.d. Do you have a list of corrupted / foreign files? Did you make a backup of those files you can share with me? (Please send me an email if you want to discuss dropping hem off.)


A precarious situation. While the practice of copying binaries over may allow you to regain (partial) control it also destroys evidence and may alert the cracker. You also create for yourself the illusion the problem is solved. But without further investigation you can not ensure 0) you have discovered all changes made, 1) the point or points of entry, 2) what data may have been siphoned off the system and 3) if the machine was used as a springboard to others or the other way around.


* BTW, any particular reason for not answering all of my questions and as detailed as I asked of you? Please understand that the more information you provide the better we can help you. Best not waste more time.

1 members found this post helpful.

Hello Unspown,
Well, I haven't taken backup of the corrupted files, I directly replaced them with healthy files. The files were corrupted on 29th Sept. 2011.
The 'last' command doen't give any clue as the logs have been deleted, but the 'history ' command gives some clues.
The suspecious part is :


155 cd /var/tmp
156 cd .x
157 ls
158 rm -rf atack.tar.gz
159 /sbin/iptables -I INPUT -p tcp --dport 55555 -j ACCEPT
.
.(the same cmd is repeated down till line no 230)
.

230 /sbin/iptables -I INPUT -p tcp --dport 55555 -j ACCEPT
231 ls
232 tar xvf ajrk.tar.gz
233 cd s
234 cd shv4/
235 ls
236 ./setup altavoces 55555
237 cd /var/tmp
238 cd .x
239 ls
240 tar xvf scanx2011.tar.gz
241 ls
242 cd ScannerX/
243 ls
244 ls -all
245 cd .s
246 ls
247 rm -rf bios.txt
248 cat trueusers.txt
249 ls
250 ./a 120.147
251 ps ux
252 killall -9 scanssh
253 ls
254 rm -rf bios.txt
255 ./a 203.184
256 s
257 ls
258 rm -rf scan.log
259 ps ux
260 kill -9 7007
261 ./a 140.110
262 ps ux
263 killall -9 scanssh
264 cd /var/tmp
265 cd " "
266 ls
267 tar cvf scanX2011.tar.gz ScannerX/
268 ls
269 rm -rf ScannerX
270 mv scanX2011.tar.gz /var/tmp/x
271 ls
272 cd ..
273 ls
274 cd x
275 cd /var/tmp/x
276 tar xvf x
277 cd ..
278 ls
279 cd /var/tmp
280 cd .x
281 cd ..
282 ls
283 rm -rf ScannerX
284 mv x scanx2011.tar.gz x
285 mv x scanx2011.tar.gz
286 ls
287 tar xvf scanx2011.tar.gz
288 ls
289 cd ..
290 ls
291 cd /var/tmp
292 cd .x
293 cd ..
294 ls
295 rm -rf ScannerX
296 mv scanx2011.tar.gz .x
297 ls
298 rm -rf " "
299 ls
300 cd .x
301 ls
302 ls
303 tar xvf nonames.tar.tar
304 cd emech/
305 ls
306 cd ..
307 ls
308 rm -rf emech
309 ls
310 tar xvf nonames.tar.tar
311 cd emech/
312 ls
313 ./pico m.set
314 ps aux
315 ls
316 ./pico m.set
317 export PATH="."
318 bash

At startup eth0 and eth1 do not start.When i start explicitely, it works but gives msg: kernel is confused and dazy.
At shutdown again the segmentation fault error is given (now its not coming while starting the services.)

I hope I have conveyed you the things.(And again it has confused me)

Interesting. This is designed to ensure that port 55555 is open in the firewall to accept inbound connections. Have you looked at the output of ps, netstat, or lsof to see if there is any application listening on these ports and to see if something is connected? Note, you will need to verify the integrity of these utilities before relying on them to give you proper information. You might consider downloading a known good, local copy, and running with that.
This and the rest of the commands reads like someone is actively connected to the machine at the time these commands are being run, rather than an executed script. The copious use of ls gives me this impression. It is also interesting that they run a command and pass it the parameter 55555, which is probably the application that is set to listen on this port.
Note the use of the tmp directory. Tmp is one of the few locations that routinely has world write permissions. I would look for additional evidence to support it, but this looks to me like a normally non privileged application, such as Apache, was used to upload files to your tmp directory. A shell prompt was then obtained. Once they had the shell prompt, they managed to gain root level privilege and I would look for evidence of how this may have occurred. They would have needed root access to replace binary files in the /bin directory. Note, however, that non privileged users can open high numbered ports, hence the 55555 in the earlier stages.
The start up and shut down controls are script files. It is possible that these were altered. It is also possible that the binaries called in these scripts have been altered.

Some of the file names in the bash history may provide a clue as to the intent. I noticed that a lot of the files are titled with the word scan. Does this mean scan as in scan your computer or use it as a platform to launch other nefarious activity? The files nonames, emech, and bios.txt are a little more nebulous but I find the word bios to be disconcerting.

...so the attacker gained root on or before that date, installed her stuff and continuied to destroy the log files. As you rebooted the machine you ensured no deleted (but potentially still open) log files would be recoverable. While it's not the time to pass judgment one could construe all of your reflexive actions so far as (albeit unintentionally) aiding the attacker.


From the names listed (accepted as no more than prima facie evidence) you've got a set of directories in world-writable /var/tmp that contain the SHv4 rootkit: the args to ./setup are the password and SSH port (and installing it requires root rights so while not explicitly stated, as you should have, I'll assert this was the shell history of root), the standard SSH IP range scanner (bios.txt being a list of addresses scanned) 'sshscan' and an EnergyMech IRC bot. Next to dropping trojaned binaries and SysV init scripts a default layout of SHv4 also contains files in library locations and libraries. While you haven't presented any listing of anomalous files the latter is most likely the cause of these segmentation fault warnings.


Posting roots shell history is nice, but it is incomplete information and it only pertains the aftermath.

Since your logs have been deleted and you have not presented more information we've exhausted things to learn from the system. If you have to make backups, say of your database, now would be the time to do so as the next task will be taking the machine offline and preparing for repartioning, creating file systems and installing and hardening the OS. Feel free to present a list of applications and exact versions to install, including whatever runs in your web stack, to receive suggestions for installing and hardening. In the meantime you should do a web search for "hardening RHEL" yourself and read docs from Red Hat, SANS, CIS and OWASP on the matter.

Hi,

I can't say how to do more to secure your server, but in case of some 'major' system modifications I have tried and installed OSSEC which instantly emails about important system files are changed , for example:
I hope it can help you somehow.

good luck

how much time and effort do you have in this build prior to being hacked?

i myself advocate a strategic approach which coincides with the resources available and the value of the system/data in question. in your case it sounds like you have little resources available and the data doesnt seem too important.

if this is your 1st attempt to provide services to the public then i'll suggest you dd if=/dev/zero all the dedicated disks on that system, and then spend some time getting educated on how-to secure the system and services you wish to offer to public access, then rebuild.

your attempts to copy binaries from one system to the other should be a tactical task in an effort to gain functional access to the OS to perhaps preserve some data.

your statement "...I had set a firewall (allowing http, ssh etc)." has me shaking my head. the word "firewall" is very generic, and you ignorantly use "etc". what exactly is allowed for "etc"??? what "firewall"?

what httpd and sshd was it? were they running as root? what does "yum --security check-update" say (or similar for whatever distro you have)?

last question. are you really wanting to know what happened?

The impression that I get from the original post is that the OP isn't entirely new to Linux, but is new to running a public facing server and received a hard lesson in security. These things can and do happen and chances are that they will learn a tremendous amount from the experience. I also get the impression that they may be coming from a Windows background based upon the firewall comment. The primary purpose of a firewall is to ensure that ports that you did not explicitly intend to open remain closed. In Windows, especially earlier versions, ports were open by default and it was necessary to have an independent application close them for you and this, along with a virus scanner, became synonymous with being secure. I take the statement "allowing http, ssh etc" to mean that they used a firewall to ensure that only the ports for public facing services were run. When running a public facing system, while being a wise precaution, this by itself is woefully inadequate in terms of security.

In this instance, based upon the very limited evidence (the command log and use of /tmp), I suspect that a public facing application was compromised to gain a bash shell which used that as a platform to gain root. Running outdated web applications and complex content management tools is a common vector for this type of attack. I would take as serious look the revision level of the distribution and the applications , which I note is information that is conspicuously absent in this thread and this gives me suspicion that the OP knows something that hasn't been shared, e.g. running an old non-patched RH system.

Before looking at a rebuild, I would ask what services do you require versus what services were you providing. Once the required services has been identified, I would investigate and build a comprehensive, layered plan to secure those services. Wiping the HDD, with the "dd if=/dev/zero" is not a bad idea as a precaution when rebuilding the system.

Thank you all for your help.
Thank you , Unspawn, once again!
And Linux Kidd,
Some of the things you said are right. I don't have many resources to share and even much data(fortunately I had taken backup of the required data just before the attack). I have not been technical while sending messages. Sorry for that.(Maybe , that was because I desparately wanted my system to get recoverd.)
I must say you are a knowlegable person, and at the same time I would say - The last question you asked doesn't give any sign of wisdom. All the best to you too...

Mahesh,
my question was valid. if you wish to learn and figure out exactly what/when/how/why of what happened you will need to invest time & $$ & people to do that. it was a question to get you to ask yourself the question "do i have the resources or can i obtain the resources to do such?". its a yes or no answer. if the answer is "no" then a wipe-&-rebuild should be the course of action. if the answer is "yes" then the machine needs to be quarantined and a proper forensics path needs to be followed. yes or no you still need to rebuild since in your case the box is too far gone.

Noway2 has offered some excellent info/advice, i'll just add that you still need invest in the resource(s) required to properly serve public accessible services. the level of security required will dictate how much resource(s) are needed, etc. perhaps start by readidng some "best practices" documentation at NIST or SANS.