Restoring data after SucKIT rootkit hacking. How can I tell what if any is safe?
 

My laptop has been hacked using SucKIT. From logs I recovered after re-installing chkrootkit and rkhunter and their config files I think I know the date and time of the attack as well as that it was SucKIT.

Is reinstalling (which includes reformatting and reloading) Lenny enough to be safe? Do I have to change partitions size or zero drive? Following reinstall chkrootkit and rkhunter have no issues.

I noticed some .doc files were renamed .doc.doc and I had strange messages when I attempted to open .html files, such as "this document is of type .html, and has a .html extension, are you sure you want to open it" I know it can add extensions to files in order to hide them.

Is there any way of knowing what data files such as .html and .doc .odt etc are unaffected and safe to restore? I would really like to recover these as there is years worth of information and work on this hard drive.
No backups have been unaffected by SucKIT. I backed up drive using dar and burnt it to DVDs. How can I safely restore information? Could I unzip backup onto an external USB hard drive safely and copy data files from there?

I just haven't found much information on restoring data files.

Cheers.

Finding SuckIT, at this point in time, is rare. Very rare. I would be interested to know your exact kernel version, how you found out (log excerpts, tool output) and made certain (autopsy as in running tools on the "dead" machine by booting a Live CD) and where what was stored. Depending on what was on your machine would determine what you should give priority to. Because somebody obviously took more than average interest in your machine. If I would pull off such a compromise I would be interested in company docs or auth for other systems, definately not stuff that would get me detected like "infecting" or changing documents.

In short: before you procede, please, more details.

SucKIT hacking
 

Thanks for the reply Unspawn. Since the first hacking attempt, this is a repeat occurrence with the same rootkit, I have been reading up a lot on security. A few weeks ago I thought I may as well just update the system as well and switched from Debian Etch to Lenny, repartitioned,etc. Currently using a Ubuntu 8.10 live DVD to access the internet.

I have wiped the drive but have backups of it. The logs showed the hacking, especially after I removed chkrootkit and rkhunter and config files and reinstalled them using synaptic.

I haven't configured or done anything on the clean drive but install Lenny. It wouldn't be big deal for me to revert back to the hacked system and supply info.

You may be right about someone taking a special interest in my laptop - disgruntled former casual employee wannabe 'partner'. Long story, was a flatmate when evicted by landlord, wanted me to open ssh ports on home network. Access to laptop. Bad news.

Happy to do a bit more forensics, but then get on with things with even more security on laptop itself, [proxy server kind of defeats the purpose of laptop]. Good news is it has improved my security knowledge in linux!

Get back to you in a few hours.

Cheers

Thanks for providing more background information. The point is these days, with the ease RFI's and such provide, using a LKM isn't as "average" an MO as it used to be. Meaning if somebody "invested" that much and took that amount of interest in your case, and this having gone undetected for some time, the offender would have had ample time to leech passwords (for other systems?), personal and financial information or other documents of interest. In other words, it is not the purely technical side of things I would worry about but where things transcend the technical and cross over to and have implications wrt Real Life.

(IANAL but having worked with lawyers before, if you can show rudimentary evidence of the intent of him stealing business ideas, code, et cetera, then some lawyers might conclude you have enough grounds to opt for preliminary custody of the offenders personal computers and storage media. The way this works is all his digital posessions will remain locked away pending forensic investigation, having the effect of a chokehold. Even if the forensic examination would be inconclusive it would certainly send a nice signal to him and inconvenience him quite a bit. Again, IANAL, so you might want to get proper advice before pursuing the idea.)

mazinoz.
unSpawn is right (as usual ;) )

You should be seriously worried about all sorts of personal data - your bank, credit card, PayPal passwords, email addresses of all your family & friends, personal information etc. falling into the hands of a potentially cunning, dishonest and ruthless person.

If I were you, I'd certainly alert my bank & the likes ("Watch out for suspicious / unusual transactions"), and also get the police involved sooner rather than later.

I have recently helped a friend with a suspected fraud at his business. The employee under suspicion had already left, but their computer was untouched. The first thing I did was not even boot it into windows, but remove the HDD, mount it as RO in linux and take a dd copy of every partition.

I was then able to replace the HDD where it belonged and begin to look for evidence by mounting the dd'd images on linux.

20minutes with some undelete utilities, and there was ample evidence to call the police immediately. I expect to be summoned to court sometime in the next year or so, but I will be able to account for my actions. I am sure the police have much better forensic analysts than I am, and I did not wish to mess up the raw data for them. But I needed something to get them interested. They only asked for the hardware about 8Mo. after the fraud was reported to them. Meanwhile it (HDD and PC) had been locked away, untouched.

The fraud I discovered is an accounting fraud, yours is possibly a personal vendetta. Either can be very unpleasant.

So, the bottom line is: Do not touch the infected HDD.

Remove it, replace it with new, reinstall to a fresh HDD.
Then take a dd copy of the infected one with it mounted ReadOnly and mount the dd'd partitions for your own forensics.

And take professional advice because IANAL.

Good luck, you need it.

Thanks everyone for your replies, it has given me a lot to think about. I know that SucKIT allows hackers to see your terminal using talk and log keystrokes. Some months ago I know HE had his identity stolen as he had mail still sent here and was keen to hear from his bank and asked me to open a letter, when it did arrive. Few weeks later my card was knocked back at grocery store, I also had had my identity stolen, but we use different banks, and I use that debit card mainly for online purchases and don't put a lot of money in it and use a number generator to do online banking and have done for years. It is possible an online pharmacy we both used had their records hacked. Banks are not forthcoming on these issues.

I will go back in the logs I have and see if I can find an IP address or anything that points to a particular person. Things have just got really bad in the last few weeks re using laptop. However I know this then may not be the hacker but another victims computer. My basic firewall lights up like a Christmas tree every day from failed portscans and connection attempts.

I know he has hacked my laptop because he knew things about what I was doing online he could only know from being able to see the screen. That was the final straw in me insisting he leave. He also placed a script on it 'mazinoz connects to internet' but when I went to open it, it disappeared and the Desktop jumped. I had been having sluggish performance but after this it really improved. Then he rang me and asked me to take him to the doctors and said he didn't expect me to after what he had done. On another occassion he said something to the effect that he should remove the rootkit as I have suffered enough.

There are extenuating circumstances here, he has a mental illness and cerebral palsy and attended a special school as a child. I knew nothing about this when he wanted to work in my business, initially on one project. Long story. However he has an electrical engineering degree, though I have not seen it. He still has stuff here that would fill a truck and his desktop computer. Tempting. If it is him and not another hacker that has done all the damage and he was not 'just' snooping, he is going to want to leave the country fast! I will also look at your suggestions. It would kill him to have his computers taken off him and impounded. But if he can be shown to have done this enough to get the police to do something I may very well try to do this. A threat of legal action may also deter him. At this point in time I want to get on with being able to safely use my laptop for work, research and educational purposes as I have received funding for a IT course. The security level is now going to be moved way beyond that of a personal computer however for the above mentioned reasons, and also I will be forced to use Windows for the course.

Restoring data after SucKIT rootkit hacking. Update!
 

Dear Unspawn

Quick update re rkhunter results. Installed unhide_20080519-2_i386.deb and skdet-1.0.tar.bz2 and ran them. After running skdet -c -s and using vim to read the README file, got a message on exiting vim that 'Can't write vim-info file /home/root/viminfo!

Ran a copy of clean install rkhunter --checkall, and again on exiting got the same message. My root prompt changed also to what it should be root@..., I have attached a screenprint of original results.

Latest log (abbreviated) are below. Just thought I'd let you know as you have a special interest in rkhunter; and results may help other Lenny users.

:)
Any informative insights also appreciated.

http://www.xs4all.nl/~dvgevers/skdet/skdet-1.0.tar.bz2
packages.debian.org/lenny/i386/unhide/download

Performing additional rootkit checks
Suckit Rookit additional checks [ OK ]
Checking for possible rootkit files and directories [ None found ]
Checking for possible rootkit strings [ None found ]

[20:57:07] Checking for Suckit Rootkit...
[20:57:07] Checking for file '/sbin/initsk12' [ Not found ]
[20:57:07] Checking for file '/sbin/initxrk' [ Not found ]
[20:57:07] Checking for file '/usr/bin/null' [ Not found ]
[20:57:07] Checking for file '/usr/share/locale/sk/.sk12/sk' [ Not found ]
[20:57:07] Checking for file '/etc/rc.d/rc0.d/S23kmdac' [ Not found ]
[20:57:07] Checking for file '/etc/rc.d/rc1.d/S23kmdac' [ Not found ]
[20:57:07] Checking for file '/etc/rc.d/rc2.d/S23kmdac' [ Not found ]
[20:57:07] Checking for file '/etc/rc.d/rc3.d/S23kmdac' [ Not found ]
[20:57:07] Checking for file '/etc/rc.d/rc4.d/S23kmdac' [ Not found ]
[20:57:07] Checking for file '/etc/rc.d/rc5.d/S23kmdac' [ Not found ]
[20:57:07] Checking for file '/etc/rc.d/rc6.d/S23kmdac' [ Not found ]
[20:57:07] Checking for directory '/dev/sdhu0/tehdrakg' [ Not found ]
[20:57:07] Checking for directory '/etc/.MG' [ Not found ]
[20:57:07] Checking for directory '/usr/share/locale/sk/.sk12' [ Not found ]
[20:57:07] Checking for directory '/usr/lib/perl5/site_perl/i386-linux/auto/TimeDate/.packlist' [ Not found ]
[20:57:07] Suckit Rootkit [ Not found ]

Performing malware checks
Checking running processes for deleted files [ Warning ]
Checking running processes for suspicious files [ None found ]
Checking for hidden processes [ Warning ]


[20:57:18] Info: Starting test name 'hidden_procs'
[20:57:35] Checking for hidden processes [ Warning ]
[20:57:35] Warning: Hidden processes found: 24622


Checking for files with suspicious contents [ None found ]
Checking for login backdoors [ None found ]
Checking for suspicious directories [ None found ]
Checking for sniffer log files [ None found ]

System checks summary
=====================

File properties checks...
Files checked: 127
Suspect files: 2

Rootkit checks...
Rootkits checked : 107
Possible rootkits: 0

Applications checks...
Applications checked: 4
Suspect applications: 0

The system checks took: 2 minutes and 43 seconds

Not out of the woods yet, but at least I have some idea now.

Cheers

First of all I'd suggest running checks from a Live CD. Deleted files (lsof) may be due to 'at' jobs. Should be easily checkable running 'lsof -P -w -n|grep dele'. Hidden processes in RKH and Chkrootkit may be due to short-lived processes. If /proc/24622/ isn't accessable running 'unhide(-linux26)' in all three modes could show clues. Could you elaborate on your OP conclusion of SuckIT? It's an init infector so running 'strings /sbin/init | grep -i fuck' should return "FUCK", pardon my french.

Sorry for not attaching it I could NOT find it earlier when posting, but found it today!

root@marilyns-laptop:~/DebianLive# cd /
root@marilyns-laptop:/# /sbin/init | grep -i fuck
Usage: init {-e VAR[=VAL] | [-t SECONDS] {0|1|2|3|4|5|6|S|s|Q|q|A|a|B|b|C|c|U|u}}


/sbin/init/ u

that I did on Saturday rendered no output.

The number of 'Warnings' decreased after this. But unhide and skdet had quite significant effects when run alone and in rkhunter.

I am currently using hard drive but will reboot to live CD tomorrow and do as you suggest. The dar backups are supposed to return the system to the same state as at the time of the backups. I know they have deleted unwanted files in the past when I restored the backup. Have also tried live-magic and live-helper to make DVD but it is about 6G and don't have materials to burn this at moment.

Cheers and once again sorry for not posting relevant .png.

Uhm, that's not what I posted you could try...


If it was SuckIT then '/sbin/init u' should uninstall it except there's no trailing slash after init: it's not a directory (typo?).


I would not trust a machine after it got infected. And without proper auditing there's no way telling what (else) got installed (and on whichever machines you have access to, to be complete). I'd love to comment on running Unhide, Skdet and RKH or combo's of those but I'm afraid it's all becoming a bit disjointed for me what got run when and where. I'll wait for your autopsy results, after that if you want to try your hand at more advanced autopsy you should have a 'dd' image of the disk (DAR isn't good enough). In your case, regardless of the fact that repeated booting of the machine after detecting the incident diminishes tracing things drastically, I would advise you should.


Back to the OP:
Recall that formatting a disk does nothing but overlay partitions with the filesystem structure. If places exist where information can be recovered (regardless of indirect addressed blocks etc, etc) information can be recovered. Practically speaking for a perp this would mean getting access to the machine one way or another to make use of it and filesize would severely limit what would be usable after recovery. Zeroing out the drive would, let's say exorcise all daemons, which from a incident handling best practice and psychological point of view is a Good Thing. Start over afresh.


Typically skilled intruders will want to keep quiet to avoid detection. I don't know what tools might cause collateral damage like this but when encountering messages from any DE/WM I would open a terminal window and run 'ls -al' or 'stat', 'file' and maybe strings on the subject to find out more in an unobscured way.


One problem I have with this is the definition of the word "safe".

Practically speaking one could easily salvage, inspect and verify (rinse, wash, repeat) human readable files for restoration ranging from configuration files to documents. Even some seemingly binary data is up for inspection. For instance OOo documents are just gzipped XML and therefore it would be hard to hide things (OK, maybe hiding in plain sight with Base64 blocks or stego in images for the truly paranoid) but easy to inspect things. This however does not (and should not) apply to system binaries, database dumps, proprietary and other data that can't be inspected otherwise. One needs independant means to verify the integrity of those. In your case verifying things with backups is harder, but given a perps interest, not impossible. (BTW, sure one could sandbox and load or execute suspect items in a guest VM for closer inspection.)

What the problem with the definition of safe is not infection of files but the implications I posted in reply #1 and what Tredegar expanded on later on: "If I would pull off such a compromise I would be interested in company docs or auth for other systems, definately not stuff that would get me detected like "infecting" or changing documents". In short: business, personal and financial information.

Dear Unspawn

Sorry about incorrect command, misread it. You have given me a lot to think about and do and I will get back to you as soon as practicable.

So how's this coming along?

No disrespect intended with getting back to you. Been battling head cold and sinus headaches, laptop issues - CMOS battery replaced still no improvement, BIOS shows main battery ok, problems booting till one or both batteries feel up to it. My ISP insist on disconnecting me at whim, (downloaded security DVD's but still have > 1G till limit). Can't login to home page to send them an email as they suspend this also. Can't phone them as they disable phone services at same time and if you phone them on landline, or other mobile SIM you are told '3 Care' are experiencing difficulties, but it has been like this for 6mths! Mobile phone battery also just died. 12mths warranty must be up!

This machine is no use for legal recourses. I would like however to find the IP of the hacker and perhaps how they got in, but also poor security practices on my part. Also how to clean data if possible.

strings /sbin/init | grep -i fuck

ran from live DVD Ubuntu8.10 returned nothing.

Mounted hda1 and ran it with same result.

Sorry for being a newbie, but if I use Ubuntu live DVD will

dd if=/mnt/hda1 of=0 work? and so on for other partitions.

I appreciate your input, but a sys admin I know has advised it is hard to get a conviction for hacking here, even with the police watching as they do the deed. Actually trying to get the police to do stuff in Qld is hard. Resources devoted to catching bank hackers and paedophiles are higher priority. No money stolen here, just a lot of timewasting and malicious damage. Would like to know who hacker was for sure though. Don't know No.1 suspect's IP address, but do know who his ISP is. However he may have used a work network or internet cafe. Have asked his ISP to monitor for connections to this laptop's IP address, sent by him, if possible. My ISP are extremely uncommunicative about everything. Can't even contact them at moment, 22mths to go on their contract.

Once again thanks. On a philosophical level, I believe securiity practices should be part and parcel at all levels of setting up networks, eg: software to run, protecting hardware, but still educators insist on just tacking it on at the end of a course, especially Windows based ones as almost an afterthought. Just not feasible these days.

Just found this interesting and informative article in case anyone is interested. Details how hack occurs.

http://www.debian.org/News/2003/20031202

No problem. But if I would want advice on my Terrago I go to my bicycle repair shop. I wouldn't ask bicyclerepairman for legal advice though if you dig what I'm saying. Not disrespecting your sysadm's vast knowledge, mind you. Anyway, it seems you've half made your mind up. I'd say ditch the IP-finding (as you won't be met with cooperation unless you've got proper legal backing), cut your losses and get on with things. Seal up your LAN, investigate connected machines, investigate or have investigated any machines you have privileged access to (revisit posts #2 and #4 for the reasoning), change of have credentials changed where needed, save the stuff you really need to keep, then zero out the drive and reinstall from scratch.

Thanks for help, pretty much what I decided to do, am under pressure to make this laptop usable at moment. Will do a bit more reading re patching for this vulnerability and then get on with it. I find this forum very helpful, not just for this problem.