A Better Way to Delete a Server

b10m3ch4 · 11-09-2009, 10:35 PM

Our company regularly uses dedicated servers for client projects. Recently we were presented with the possibility that data we deleted off a dedicated server before ending the lease was somehow recovered from hard drive of that machine. Our end is secure, and the client swears that data was not leaked from their firm. We have a script we execute that deletes all the log files, and data associated with client projects, but apparently it’s not solid enough. We deal with middleware between financial institutions, so it’s rather important that the IPs of the machines that previous connected to our discontinued server, remain private.

Now, here’s our silly script. I’d appreciate any input on improving. I realize it’s a long way from perfect, but up until recently we had no reason to distrust hosting companies. Moving forward, we just want to be as secure as possible. I have taken out some lines of code that represent the removal of specific applications, otherwise this is as-is. What’s needed is the means to completely and securely erase all data from a remote machine once we have discontinued its use. I am sure other users may have similar needs.

Code:

#!/bin/sh

rm -fR /var/logs/ssl_request_log
rm -fR /home/admin/domains/TheDomain.net/logs/*.*
rm -fR /var/log/directadmin/*
rm -fR /var/log/exim/*
rm -fR /var/log/proftpd/*
rm -f /var/log/secure*
rm -f /var/log/maillog*
touch /var/log/maillog
rm -fR /var/log/httpd/domains/*
touch /var/log/httpd/domains/TheDomain.net.error.log
rm -fR /var/log/httpd/error_log
touch /var/log/httpd/error_log
rm -fR /var/log/messages*
touch /var/log/messages

chrism01 · 11-09-2009, 10:39 PM

Strictly speaking, rm just removes inode entries, not actual data; see photorec amongst others for recovery.
What you need is shred http://linux.die.net/man/1/shred or dban http://www.dban.org/

b10m3ch4 · 11-09-2009, 11:19 PM

I tried shred, and I actually use it, the problem is it needs to be able to hit a file that actually exists, it's not as forgiving as rm. I can't shred * in certain directories as I'd need to. Here is an example of another "logic bomb" or sorts we have tried, but it just gets the log files, and depends on an updatedb being fresh.

Code:

#!/bin/sh
mkdir ~/shred
cd /var/log
mv *log* ~/shred
cd ~/shred
for i in `locate *log*`; do
shred -vzu 8 $i;
done

If I could modify this script to attack all files created or modified from X Date to Y Date, that would be perfect, but I am fairly new to shell scripting, so I have had no luck with getting the above script to do anything more than attack files with the .log extension.

chiragrk · 11-10-2009, 12:33 AM

There are two questions you need to ask yourself here:
- Am I worried that I missed deleting some files? Files which may still have some amount of client info?
- Am I worried that that the files I've deleted can be recovered by someone who has access to the server?

The concern #1 above can be solved - painfully though by auditing the system thoroughly for the applications/softwares used. For starts you seemed to have missed /var/log/btmp and /var/log/wtmp which hold information about who logged in remotely.
The concern #2 will definitely require some advanced techniques.

pcunix · 11-10-2009, 07:18 AM

How secure is secure?

The shred man page itself shows that you may well still be vulnerable and so does this page:

http://www.cs.auckland.ac.nz/~pgut00...ecure_del.html

If the machine is "inactive", why can't the drive be physically destroyed? Because it's a lease, obviously. And that's your answer: if the data is this important, you shouldn't be leasing machines - or at least should be putting your own disposable drives in.

salasi · 11-10-2009, 03:52 PM

Well, rm clearly won't work (data easily recoverable) at least without using rm in combination with something else. It is also vulnerable to the problem that you might get one box with different files than others and the files on that get overlooked. This could be unpleasant.

Shred only shreds files and that is probably not good enough (on its own) either because of the problem on the man page or because you might overlook files.

I didn't see anything on the dban site that suggested that it met any particular data destruction standards.

It seems to me that you have not only to remove the information that you had, but you need to overwrite and erase the whole disk several times. Only a single pass of overwriting is vulnerable to forensics experts and that seems like something you don't want if the data could have a value in excess of the recovery cost.

Maybe you should pass this problem on to professionals. Maybe you should take your disposable disk and just put it in your safe (that has other advantages, too, but the safe may not be safe enough). Maybe you should use a sledgehammer, which may be the most satisfying solution.

There is a brief description of data destruction standards here:
http://www.mcdpri.com/mcdpri/data-de...-standards.htm

smeezekitty · 11-10-2009, 04:09 PM

since the lease is over why don't you dd it?
for example dd it with zeros then with random
then with zeros again. that would definitely destroy all data on the disk to the point where it would cost more to recover part of the data then the data is likly worth.

unSpawn · 11-10-2009, 04:18 PM

Quote:

Originally Posted by salasi

It seems to me that you have not only to remove the information that you had, but you need to overwrite and erase the whole disk several times.

Using a 'rm' script is bad, especially where transactions between financial institutions are bound by strict data safety regulations. So I agree with complete disk erasure. But not only that: you also need to verify things afterwards. Only when you audit you can actually ensure medium sanitation.

Quote:

Originally Posted by salasi

Maybe you should pass this problem on to professionals.

Excellent advice. Not only can you hand off the whole sanitation process but (certified) companies can also provide you with complete process reporting so you got everything covered in one fell swoop.

b10m3ch4 · 11-10-2009, 11:07 PM

How do I 'dd' anything when my only access is SSH? To answer the question about as to why we are leasing, it's frequently necessary to establish a point of presence between our development location and the client location. That, and the software itself is setup to be ran on any machine, so it's necessary from that angle to for proof-of-concept. So my question now is, what's the best I can do remotely, shred would work, but I need a script to find files to destroy apparently. I like the DBAN idea, but how do i do that remotely?

And actually, we are moving towards setting up a TrueCrypt virtual drive on each box, then just deleting that when we are done. That protects client data as good as we can on our/their budget. However, we still need to kill every possible log file of any communication with the box that would show an outside IP.

I have added this to the script, and it works well.

Code:

shred -vzfu 8 /var/log/btmp
touch /var/log/btmp
shred -vzfu 8 /var/log/wtmp
touch /var/log/wtmp

pcunix · 11-11-2009, 06:32 AM

I like the encryption idea. That should do it.

funkflex2004 · 11-11-2009, 12:02 PM

I had to do something similar to wipe the hard drive as much as possible.

I used a LiveCD version of linux eg Ubuntu so that the hard drive i wanted to wipe was not in use. (Could rescue boot partition/CD be used?)

I looked for the hard drive I wanted to wipe and used "shred" with a minimum of 4 pass random wipe.

Code:

shred /dev/sdd -f -v -z --iterations=4

The more passes the more difficult and expensive it is to recover the data.

Question: is shred similar to dd?? or better still does shred use dd??

catkin · 11-11-2009, 12:09 PM

Quote:

Originally Posted by salasi

Only a single pass of overwriting is vulnerable to forensics experts ...

Has this ever been reliably demonstrated to allow recovery of significant data?

catkin · 11-11-2009, 12:11 PM

Quote:

Originally Posted by funkflex2004

Question: is shred similar to dd?? or better still does shred use dd??

If it is effective it needs to do more than dd because recoverable data may exist in bad blocks that dd cannot reach.

kschmitt · 11-11-2009, 01:38 PM

Umm.

Is this a dedicated physical or virtual server? If it's a dedicated virtual, your SOL. Nothing that _you_ can do will guarantee it's wiped.

If it's physical on the other hand, using rm won't remove the files to the point you want. If someone snagged your drive there are plenty of utilities that can be used to recover the removed files, or parts of them. Heck, if it's ext2/3 or fat you can use grep to get interesting bits of "removed" data off the drive! In case you were wondering, dding over a file isn't a sure fire way of removing it: you need to use it on whole partitions/volumes, if not whole drives. I'm not sure how shred works.

As to only have ssh access, you could do any one of the following really dangerous things:
1) Put an entry in lilo.conf/grub.conf that calls the program to wipe the drive. Set that as the default boot option
2) Create an initrd image that wipes the drive. Install it.
Or my favorite insane trick
3) Carve off a new volume in LVM, then install a tiny util-distro (with the same ip address & a username/password for yourself) on it, change grub.conf/lilo.conf to boot to that distro, then do the wipe from there.

Seriously, _very_ dangerous things, but consider #3 if you don't have physical access, as you will have some sort of working system to re-run a wipe, test if the data is really gone, etc.

Good luck. You're going to need it, but it sounds like fun anyway.

b10m3ch4 · 11-11-2009, 02:58 PM

Quote:

Originally Posted by kschmitt

As to only have ssh access, you could do any one of the following really dangerous things:
1) Put an entry in lilo.conf/grub.conf that calls the program to wipe the drive. Set that as the default boot option
2) Create an initrd image that wipes the drive. Install it.
Or my favorite insane trick
3) Carve off a new volume in LVM, then install a tiny util-distro (with the same ip address & a username/password for yourself) on it, change grub.conf/lilo.conf to boot to that distro, then do the wipe from there.

Seriously, _very_ dangerous things, but consider #3 if you don't have physical access, as you will have some sort of working system to re-run a wipe, test if the data is really gone, etc.

Good luck. You're going to need it, but it sounds like fun anyway.

All good ideas I am looking forward to trying. Also, I am experimenting with replacing the /var/log with a sym link to a TrueCrypt partition. If this is successful, is there any other place data concerning IP connections could be stored that are not addressed in my scripts above?

Also, what little distro do you recommend, and how do you set the user and IP info of a such a distro?