LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices

Reply
 
Search this Thread
Old 06-02-2004, 10:01 AM   #1
Asiana
LQ Newbie
 
Registered: Jun 2004
Posts: 2

Rep: Reputation: 0
Server was compromised, need help


On sunday morning, someone accessed my server through one of the user accounts. They compromised root and made some major changes to include .bash_history -> /dev/null. I've tried to delete this file and make changes but nothing seems to work.. less, more, cat, vi, vim.. all rendered useless in a read-only environment.

I was able to login to the server both as my user name and as root. I removed the user which was compromised. However, the entire files are read only and issuing an input/output error.

example:
ls: .bashrc: Input/output error
ls: .vimrc: Input/output error
ls: .viminfo: Input/output error

I wish to run an fsck however I am unable to umount the drives:

bash:/home/user# umount /dev/hda1
-su: /bin/umount: Input/output error

Any suggestions? Or ideas on where I should look?

Last edited by Asiana; 06-02-2004 at 10:06 AM.
 
Old 06-02-2004, 11:07 AM   #2
iainr
Member
 
Registered: Nov 2002
Location: England
Distribution: Ubuntu 9.04
Posts: 631

Rep: Reputation: 30
Boot from a CD (e.g. Knoppix or other Live CD) to have a look around. Be ready to reinstall if you aren't confident that you'v got everything back the way it should be.
 
Old 06-02-2004, 11:15 AM   #3
Asiana
LQ Newbie
 
Registered: Jun 2004
Posts: 2

Original Poster
Rep: Reputation: 0
Is there a way to handle this without going to the datacenter and booting into a cd?
 
Old 06-02-2004, 12:39 PM   #4
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,261
Blog Entries: 54

Rep: Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841
Please first read the LQ FAQ: Security references, post 1, under "Compromise, breach of security, detection". If you don't know anything but want to take action *now* (your choice), then at least read:
Intruder Detection Checklist (CERT): http://www.cert.org/tech_tips/intrud...checklist.html
Steps for Recovering from a UNIX or NT System Compromise: http://www.cert.org/tech_tips/root_compromise.html
Formatting and Reinstalling after a Security Incident (SF): http://www.securityfocus.com/infocus/1692
...and your choice of threads searching the LQ Linux - Security forum searching for the keyword "compromise" or "hacked".


They compromised root
Clues from the system and application logfiles and files (tarballs, suid binaries)?


I've tried to delete this file and make changes but nothing seems to work.. less, more, cat, vi, vim.. all rendered useless in a read-only environment.
Meaning you remounted all disks readonly?


I was able to login to the server both as my user name
...which means it was still up and running. If it was, and the HD's where not mounted readonly directly after you noticed the compromise, this means running processes corrupted any "evidence".


However, the entire files are read only and issuing an input/output error.
Could be failing HW. Don't reboot or risk loosing data.


I wish to run an fsck
BAD THING TO DO if you want to find out more about the compromise.


however I am unable to umount the drives:
bash:/home/user# umount /dev/hda1
-su: /bin/umount: Input/output error
Any suggestions?

Dropping to runlevel 1 then remounting the disks readonly.


Is there a way to handle this without going to the datacenter and booting into a cd?
Yes and no, the decision is yours. It depends on what you're willing to trade off. If you want to trade in any forensics for just a quick glance before reinstalling from scratch (restore only if you exactly know what youre doing), then kill all processes that aren't necessary or outright suspected (lsof), disable all unnecessary users, reinstall sshd on a high port (dump the config first on the system) then run Chkrootkit, Rootkit Hunter, package manager in verify mode or file integrity checker. If you're done or if you don't want to learn more, at least (have them) save application version info, the system and application logs, auth files, contents of dirs (accessable by unprivileged users), after running mentioned apps. Then repartition, reformat and reinstall from scratch, unless you can do a restore with a backup which contents can be verified.

If you OTOH want to learn more, your first task will be to (order the colo ppl to) preserve any "evidence" left by any means. In the best circumstances this means killing the box cold (forced shutdown), mounting the disks in another box and making dd images of the full disk. If you're not on a shared host, remounting disks read-only and making dd images of all disks could do. Elif this is a shared hosting setup, just make dd images of all disks and hope for the best. In any case this way you'll have (some form of) disk image to do forensics on.


HTH.



Boot from a CD (e.g. Knoppix or other Live CD) to have a look around. Be ready to reinstall if you aren't confident that you'v got everything back the way it should be.
How can you make 100 percent sure a server is "restored" "back the way "it should be""?
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Server compromised, system files changed newlinuxnewbie Linux - General 4 10-11-2005 03:22 PM
Server Compromised? stlyz3 Linux - Security 6 09-07-2005 04:28 PM
Gentoo's server compromised? Couldn't be, right? jon_k Linux - Security 1 06-12-2005 06:46 PM
Apache server compromised? lacerto Linux - Security 3 04-13-2005 03:26 PM
Server Compromised. Pls help. phumes Linux - Security 5 08-24-2004 11:47 AM


All times are GMT -5. The time now is 04:33 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration