LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
LinkBack Search this Thread
Old 09-29-2012, 06:52 PM   #1
cmnorton
Member
 
Registered: Feb 2005
Distribution: Ubuntu, CentOS
Posts: 585

Rep: Reputation: 35
Can ASCII Files Damage A Linux File System?


We are going to wipe a Centos 5.x system clean, and install it again. The system was infected with what we believe to be a root kit. We already know most of the advice about giving up and re-installing. We are trying to recover
ASCII text files using Knoppix.

The server is about a ten year old Dell PowerEdge. We believe it is running USB 1.0/1.1. The only way we can
copy configuration files off is to a thumb drive formatted with ext3.

At the same place in our copying we saw one html file cause the copy -- rsync -- to hang, and when we either rebooted the system with Knoppix or pulled the drive out and tried to remount the thumb drive, the mount hung. Examining the thumb drive on another system running Knoppix also resulted in a hung mount command.

So, the question is are there any file settings that might damage an ext3 file system? I remember there being chip problems about a decade ago, when USB 2.x devices (our current thumb drives) were used with native USB 1.1, and I think I/O is the problem.

However, to get this system to the point where some of the virus damage was mitigated, we had to clear attributes on files like grep, mv, cp, and so on, because those programs' attributes were set in such a way that root could not delete the files (move a good copy of, say grep, into /usr/bin.

Any thoughts, other than wipe the system clean?

Thanks.

Last edited by cmnorton; 09-29-2012 at 07:25 PM. Reason: Remove solution. It wasn't a solution.
 
Old 09-29-2012, 08:38 PM   #2
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 11,794

Rep: Reputation: 923Reputation: 923Reputation: 923Reputation: 923Reputation: 923Reputation: 923Reputation: 923Reputation: 923
Quote:
Originally Posted by cmnorton View Post
... and when we either rebooted the system with Knoppix or pulled the drive out and tried to remount the thumb drive, the mount hung.
Yanking the system and/or target drive is almost guaranteed to trash the filesystem.
If you really think that file is "dodgy", exclude it from rsync, and let the copy (totally) finish. Issue a umount to ensure all the writes are actually in the USB before pulling it out. If it really is USB 1[.1] it could take a while, so let it run.
 
1 members found this post helpful.
Old 09-29-2012, 08:42 PM   #3
cmnorton
Member
 
Registered: Feb 2005
Distribution: Ubuntu, CentOS
Posts: 585

Original Poster
Rep: Reputation: 35
Also happened without yanking

It is hard to say what happened, but the I/O hung, and we could not even get to another terminal prompt to examine. There were supposed to be backups for this system, but... Thanks for your answer.
 
Old 09-29-2012, 08:55 PM   #4
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 11,794

Rep: Reputation: 923Reputation: 923Reputation: 923Reputation: 923Reputation: 923Reputation: 923Reputation: 923Reputation: 923
I was presuming you could reboot a Knoppix (or whatever) and redo the rsync. Do a mkfs on the USB, then redo the rsync (maybe) excluding that suspect html file. As I said, make sure it doesn't get interrupted next time.
If you need what's (now) on the USB, that could be way more difficult.
 
Old 09-30-2012, 08:37 AM   #5
cmnorton
Member
 
Registered: Feb 2005
Distribution: Ubuntu, CentOS
Posts: 585

Original Poster
Rep: Reputation: 35
Sync hung under Knoppix

The Sync hung under Knoppix, which leads me to believe this is a hardware/chipset issue. There were problems with the handful of chipset manufacturers that implemented native 1.0/1.1. There were device driver - level patches on Windows.
 
Old 09-30-2012, 11:04 AM   #6
unSpawn
Moderator
 
Registered: May 2001
Posts: 26,519
Blog Entries: 51

Rep: Reputation: 2599Reputation: 2599Reputation: 2599Reputation: 2599Reputation: 2599Reputation: 2599Reputation: 2599Reputation: 2599Reputation: 2599Reputation: 2599Reputation: 2599
Quote:
Originally Posted by cmnorton View Post
We are going to wipe a Centos 5.x system clean, and install it again. The system was infected with what we believe to be a root kit.
Interesting. Any nfo you'd want to share? (You're invited to email me if you would want to but not in public.)


Quote:
Originally Posted by cmnorton View Post
The only way we can copy configuration files off is to a thumb drive formatted with ext3.
That doesn't say what situation you're in, which options you've got and which ones you have explored. Speed and error-wise I would prefer booting either KNOPPIX or HELIX (MD5 93a285bfa8ab93d664d508e5b12446d3) and then first cloning the disk(s) (or partitions if the server had a server layout) to a file on a clean, physical disk, external media or another network location (else would taking the disk(s) out and cloning them using another machine be an option?) because having a 'dd' image frees up the machine so you can speed up re-installation and having a 'dd' image could avoid issues you may decide to deal with later on. Tools like Photorec, Testdisk, linen, etc, etc don't need mounted file systems but just /dev partition names or disk image file names avoiding problems due to file system inconsistencies or mount errors. Read-only access to your disk image would also prevent tainting or modifying it beyond salvage, so if you must mount Ext file systems then at least use "-o ro,norecovery,noload" to prohibit replaying and loading journal entries.
 
Old 09-30-2012, 11:21 AM   #7
cmnorton
Member
 
Registered: Feb 2005
Distribution: Ubuntu, CentOS
Posts: 585

Original Poster
Rep: Reputation: 35
More Information

Here are more details.

We have booted the PowerEdge with Knoppix.
These problems also originally occurred
when CentOS 5.x was booted and we tried
copying to the stick. After that we
only used Knoppix to boot and perform the copy.

We've formatted with ext3 a 16GB thumb drive.
We've got two of them, and have formatted them
the same way.

The PowerEdge's native USB is probably USB 1.1

Once the system hangs on copy, we've either
yanked the stick or force powered down the system.
Either way, we cannot mount the volume after bringing
the system back up. I can understand why yanking the
stick would be a problem, but not a forced power down.

That's all the information there is.

You might be wondering who would create a DMZ-d server
with no backup. It isn't us. We were called in to rescue
the Mailman distro lists. That's all we care about.
 
Old 09-30-2012, 11:43 AM   #8
unSpawn
Moderator
 
Registered: May 2001
Posts: 26,519
Blog Entries: 51

Rep: Reputation: 2599Reputation: 2599Reputation: 2599Reputation: 2599Reputation: 2599Reputation: 2599Reputation: 2599Reputation: 2599Reputation: 2599Reputation: 2599Reputation: 2599
Quote:
Originally Posted by cmnorton View Post
That's all the information there is.
OK. Let's resort to simple questions then.

- How many disks does the server have and what's their size?
- Or, if the server has a server-like partition layout, what is the size of the partition you're trying to get access to?

- Would backing up over the network be an option?
- And if so, if you boot the machine with KNOPPIX or HELIX (don't need a GUI, just multi-user and networking),
- can you get a DHCP lease and
- are you allowed to use their facilities (a machine with an unprivileged account, ample free disk space)?
- Can you 'dd' partitions over the network using netcat or SSH?

- If none of the above, if you boot the machine with any CDROM that has testdisk on it, have testdisk (run it with the "/debug /log" args) access the partition your configuration files are on, can you "explore" the directories you need access too? Are these directories empty or not? Does using the "show deleted files" toggle show deleted files?
 
Old 09-30-2012, 12:26 PM   #9
cmnorton
Member
 
Registered: Feb 2005
Distribution: Ubuntu, CentOS
Posts: 585

Original Poster
Rep: Reputation: 35
More info

I appreciate your attention to this question; we're muddling along so far by having throttled down the copying.

To answer your questions:

Backing up over the network isn't really an option.

The server is in a DMZ.

The system has a RAID, and we mount that using Knoppix.

We do get a DHCP lease, and so far, by throttling back
the copy, the sticks have not frozen.

Initially, I was wondering what might be locking up the copy; thought perhaps some attributes had been set on the ASCII Mailman files -- there were not any set -- and perhaps that was locking up the memory stick.

We may never know what causes the lockup, but as I said above, throttling down how much gets copied seems to be working.

Once we get all the data off, we'll wipe the server clean and re-install CentOS. Then, we'll set up rsync; we can ssh into the server. We're also going to set up a mirror server with rsync, so if something goes wrong again, we'll have not only a data backup, but a usable server.
 
Old 09-30-2012, 02:14 PM   #10
jefro
Guru
 
Registered: Mar 2008
Posts: 10,246

Rep: Reputation: 1255Reputation: 1255Reputation: 1255Reputation: 1255Reputation: 1255Reputation: 1255Reputation: 1255Reputation: 1255Reputation: 1255
I'd have brought a pc or laptop to use to copy over the network. At usb 1.x you will be there forever.

I'd also run some diags on this. Almost any live cd should be able to be used. If you like Knoppix then fine but the system hanging is odd. The issue could be simply hardware problems.

I have never heard of any proven means that a true only ascii file could compromise a system. An ascii file could in theory contain non-ascii and then could possibly do harm. There are only a few examples ever reported on that and none have been very harmful.
 
Old 09-30-2012, 03:00 PM   #11
unSpawn
Moderator
 
Registered: May 2001
Posts: 26,519
Blog Entries: 51

Rep: Reputation: 2599Reputation: 2599Reputation: 2599Reputation: 2599Reputation: 2599Reputation: 2599Reputation: 2599Reputation: 2599Reputation: 2599Reputation: 2599Reputation: 2599
That is one of the reasons why tools like dd are preferred. They don't have any concept of file systems and they don't need to rely on the kernel or user land utilities interpreting a file system's state or the state of its contents nor would they alter the block device or its contents. Sure you need some form of translation to get from a single RAID member to a unified block device you can access but otherwise the problem with basically everything above the basic block device level is that it adds complexity and errors. Using 'dd' or an equivalent means avoiding all of that.
 
  


Reply

Tags
ext3fs, rootkit, usb 1.1


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Can `cat` binary file damage any files on HDD? Mr. Alex Linux - General 7 05-25-2012 05:37 AM
Possible file-system damage? hacker supreme General 11 02-03-2008 10:36 AM
find a string in all ASCII files of a system markraem Linux - Software 5 12-12-2007 04:08 AM
File system damage ext mandrake 10.0 and VMware gaelen Mandriva 1 04-09-2005 11:09 AM
File recovery after partition table damage - XFS file system gracecourt Linux - General 2 01-13-2004 03:53 PM


All times are GMT -5. The time now is 12:08 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration