Hello all.

The synopsis:

Mandrake 10.1
running sshd on a non-standard port with only key based authentication
-Periodically I'll ssh or sftp in while I'm away to access my files. Using ssh and konqueror with sftp.
-security level is set to high.
-I use firestarter as my firewall.

One evening I began experiencing a number of bizarre errors while I was doing some fairly basic web surfing and email. Basically I was getting errors when I called on symlinks I had in my home directory saying that they could not be found. It was late so I left it for another day to be solved. The next morning I logged out but left the box up and running all day. When I returned home and logged in my profile had reset to default and it was as though I was logging into my user account for the first time. Imaging my surprise when I navigate to my home directory to find that every file I had is now gone, there's a catch though; all my hidden files (all files preceded by a period) are still present with their original information.

My home directory is mounted on a separate ext3 partition /dev/hdb7 . A df unfortunately confirmed my assessment:

/dev/hdb7 34G 151M 34G 1% /home

I've looked through my logs; auth, messages, security and I haven't found anything out of the ordinary. The high level of security had led to a few complaints about world-readable files in my home directory but thats about it.

Although I don't believe I've been rooted, I would suppose its not impossible.

I had accessed the machine via ssh and sftp during the day before I experienced this problem but I don't recall any errant rm -rf commands, none at all to be precise, and my perusal of my .bash_history confirms this. Furthermore I looked through all the .bash_history commands and found no reason to believe this was an issue from someone who had physical access to the machine.

The machine has already been rebooted since this event but had the same errors before it was restarted.

Has anyone seen anything like this before? Could the high level of security led the system to take action (i.e. deleting) the world-readable files it saw in my home directory? (note: not all files were world-readable but all of them are inextricably missing). The presence of the hidden files is what makes this all the more strange, I'm wondering if some failure in a ssh/sftp connection led to a haywire command being sent to my machine deleting its files.

Thanks for your input.

Edit: more descriptive subject.

update:

chkrootkit didn't turn up any intrusion signs.

At this point there doesn't seem to be any evidence of a security intrusion, furthermore rooting a box only to delete the users home directory seems a fairly pointless exercise. I can only conclude some flawed command or misbehaving program recursively deleted all non-hidden files.

If anyone has any suggestions on ext3 file recovery, I'm all ears.

Thanks for the reads anyway all.

So long as you haven't done much in those directories to overwrite the deleted files, they should still be there. Ext2-based filesystems (ie: ext3) have an undelete capability.

Check this link out:
http://www.ists.dartmouth.edu/classr...very.v0.81.php

Hopefully that should do the trick.

dping: Thanks for the advice, although it seems it may not work.

According to documentation I've found on the web it appears that ext3 file systems are not capable of "undeleting".

Check out the 7th question down: http://batleth.sapienti-sat.org/proj.../ext3-faq.html

Unless I'm somehow mistaken.

I think the real problem here is that I've failed to make a backup of my system despite knowing full well that I should.The next project I'm going to tackle is an external hard drive, its that or end up here again.

Thanks again.

Indeed. It appears googling turns up quite a bit of information on the topic.

I couldn't get to the link you posted; but for interested folks who stumble across this thread later, this info is supposedly from that faq:
Makes sense to me. Sorry for the missinfo.

The plot thickens....

It actually happened again. All files were deleted from my home directory (all non-hidden ones). I logged out and back in again and kde created a Desktop and tmp folders. These files were present for around 24 hours and then they mysteriously dissapeared again. During this period I did not use ssh or sftp to remotely access the machine. Nor did I perform any file deletions myself.

Both chkrootkit and rootkit hunter failed to turn up anything of significance. I'm doubting that this is an outside attack, furthermore this is very unlikely to be a local attack. At this point I'm going to leave all the present Desktop and tmp folders in my home directory and allow the system to sit for 24 hours and see what happens.

The only significant thing I can remember changing prior to the onset of this problem would be changing the default security level from "normal" to "high".

It sounds very much like it might be msec. You might want to actually try lowering the security level again or at least change the umask your user is creating files with. Also take a look at the msec log in /var/log/security and /var/log/security.log for anything about those files.

There is mention of my home directory files in /var/log/security.log:


Thu Mar 31 04:08:09 EST 2005
**At this time the security level was set to "normal"**


Security Warning: World Writable files found :
- /home/ghop/Desktop/Trash/New Folder_3/blue-bend.jpg
<snip>lists all home directory world writable files</snip>

The logs indicate that these files were being complained about for over a month. During this month no files disappeared.



Fri Apr 1 04:07:27 EST 2005
**On This day I changed the security level to "high"**

Security Warning: Change in World Writable Files found :
- No longer present writable file : /home/ghop/Desktop/Trash/New Folder_3/blue-bend.jpg
<snip>proceeds to list all files that were previously world writable that are now missing</snip>



Its worth noting that msec complains for over a month about other world writable files in various other directories that are still present now and still being complained about (i.e. the only files that were world-writable that were deleted were the ones in my home directory).

I must admit this is very perplexing, it seems unlikely that a security program would take it on itself to delete files without further input. I've studied the settings and configuration files for the current security level of msec and there is no way to modify how it handles world-writable files, you can only change if it checks for them regularly or not.

Unfortunately the msec documentation is horrendous. At the higher security levels it does perform some pro-active measures. I was aware that it would automatically change permissions and revert changes made to config files, but I can't say for sure if it deletes files like you're seeing (though it wouldn't surprise me at the PARANOID setting). I'd try reducing the level overnight and see if it still deletes those files. Might also want to try creating a test user (don't name it 'test') with some files in it's home dir, just to make sure it isn't something user-specific.

I created another user foo and created a world writable file in their home directory. Also I created a world writable file in my own home directory. The system report only runs at 4:07 AM (eastern standard time) so I may have to wait until tomorrow for a full test of this setup.


I called msec from the command line as root and got this in my /var/log/messages:

Apr 4 08:27:33 cerberus msec: ### Program is starting ###
Apr 4 08:27:33 cerberus msec: Reading local rules from /etc/security/msec/level.local
Apr 4 08:27:33 cerberus msec: Allowing chkconfig --add from rpm
Apr 4 08:27:33 cerberus msec: Setting password maximum aging for new user to 99999
Apr 4 08:27:33 cerberus msec: Setting password maximum aging for root and users with id greater than 500 to 99999 and delay to -1 days
Apr 4 08:27:33 cerberus msec: got current maximum password aging for user root with command 'LC_ALL=C /usr/bin/chage -l root'
Apr 4 08:27:33 cerberus msec: got current maximum password aging for user ghop with command 'LC_ALL=C /usr/bin/chage -l ghop'
Apr 4 08:27:33 cerberus msec: got current maximum password aging for user foo with command 'LC_ALL=C /usr/bin/chage -l foo'
Apr 4 08:27:34 cerberus msec: Forbidding autologin
Apr 4 08:27:34 cerberus msec: Allowing reboot to the console user
Apr 4 08:27:34 cerberus msec: Writing config files and then taking needed actions
Apr 4 08:27:34 cerberus msec: Fixing owners and permissions of files and directories

This did not generate the entries in /var/log/security.log so I may wait for tomorrow for a full report.
Even though one line reads "Fixing owners and permissions of files and directories" there is currently no change in the presence or permissions in either of the world writable files in either users home directory.


According to the documentation the "high" level of security should have the following measures:


****************************
Security level 3 ( Aka more secure system ) :

- Global security check
- Permissions check
- Suid root file check
- Suid root file md5sum check
- Suid group file check
- Writable file check
- Unowned file check
- Promiscuous check
- Listening port check
- Passwd file integrity check
- Shadow file integrity check
- Warning in syslog
- Warning in /var/log/security.log
- rpm database checks
- send the results of checks by mail if they aren't empty

- umask is 022 ( user = read,write | group = read | other = read )
- Normal file permission.
- X server listens to tcp connections.
- All system events additionally logged to /dev/tty12
- Some system security check launched every midnight from the ( crontab ).
- no autologin

- home directories are accessible but not readable by others and group members.

****************************
What's confusing is the two lines:

"- umask is 022 ( user = read,write | group = read | other = read )"
and
"- home directories are accessible but not readable by others and group members."

Don't these two conflict?

Capt_Caveman is on to something in regards to the horrible (or lack there of) of msec documentation. At this point I'm going to let the system sit for another 24 hours and then check the results.

Update:

Created new user foo and placed world writable file in their home directory. Added world writable file to my own home users directory. The msec scripts ran their nightly check at 4:07 AM, the report mentions both new files but does not delete either of them, here is the resulting log (/var/log/security.log)


*** Diff Check, Tue Apr 5 04:07:28 EDT 2005 ***


Security Warning: Change in World Writable Files found :
- Newly added writable file : /home/foo/test_file
- Newly added writable file : /home/ghop/test_file


*** Security Check, Tue Apr 5 04:07:29 EDT 2005 ***


Security Warning: World Writable files found :
- /home/foo/test_file
- /home/ghop/test_file
<snip>proceeds to list other world writable files on hard drive</snip>


Its possible that this problem is not msec related. Although I'm considering that it may wait a predetermined number of security checks before deleting world writable files. I'm going to leave the system in its present state for the next few days and see what turns up.

It may be worth noting that my home directory is mounted on its own partition /dev/hdb7 and my root file system is mounted on a separate partition also /dev/hdb5. No world writable files have turned up missing/deleted from the root file system (i.e. /dev/hdb5) but their are certainly several present there. I considered some variety of hard disk error or partition table snafu but the presence of my hidden files in my home directory despite the other files being deleted seems to refute that theory.

The problem was not with msec or the security settings at all. The actual root of the problem was with Smb4K - a windows network share browser. Here's how it happened:

When you mount a windows share through Smb4K by default it will mount it in a directory called smb4k in your home directory. So the path will read /home/<user name>/smb4k/<share name> . In my infinite wisdom I wanted to save myself from having to type in the smb4k portion of the path name when navigating to the mounted shared files and choose to change the mount directory to simply /home/<user name>/<share name> .

Furthermore I decided to have Smb4K delete the empty directories once I had unmounted their contents and closed Smb4K. So I checked the box in the GUI settings dialog that read "Clean up the default directory on exit." . Naturally hindsight is 20/20 and this change would have seemed questionable to any more intelligent person who realized that I had commanded Smb4K to clean up my HOME DIRECTORY, but not to this intrepid soul.

Interestingly Smb4K only seemed to "clean up the default directory" when I closed it with shares still mounted (its supposed to unmount all shares if you just close it).

Oh well, at least I figured it out why it happened I'm sure others have seen more perplexing problems and never found a solution.

Moral of the story: MAKE BAKCKUPS!
and try to avoid convenient settings like "Clean up the default directory on exit." "corrupt hard drive" or "print /usr/src/linux" unless you're fairly conscious of what you're doing.

Interesting. Nice detective work