Can ASCII Files Damage A Linux File System?
 

We are going to wipe a Centos 5.x system clean, and install it again. The system was infected with what we believe to be a root kit. We already know most of the advice about giving up and re-installing. We are trying to recover
ASCII text files using Knoppix.

The server is about a ten year old Dell PowerEdge. We believe it is running USB 1.0/1.1. The only way we can
copy configuration files off is to a thumb drive formatted with ext3.

At the same place in our copying we saw one html file cause the copy -- rsync -- to hang, and when we either rebooted the system with Knoppix or pulled the drive out and tried to remount the thumb drive, the mount hung. Examining the thumb drive on another system running Knoppix also resulted in a hung mount command.

So, the question is are there any file settings that might damage an ext3 file system? I remember there being chip problems about a decade ago, when USB 2.x devices (our current thumb drives) were used with native USB 1.1, and I think I/O is the problem.

However, to get this system to the point where some of the virus damage was mitigated, we had to clear attributes on files like grep, mv, cp, and so on, because those programs' attributes were set in such a way that root could not delete the files (move a good copy of, say grep, into /usr/bin.

Any thoughts, other than wipe the system clean?

Thanks.

Yanking the system and/or target drive is almost guaranteed to trash the filesystem.
If you really think that file is "dodgy", exclude it from rsync, and let the copy (totally) finish. Issue a umount to ensure all the writes are actually in the USB before pulling it out. If it really is USB 1[.1] it could take a while, so let it run.

Also happened without yanking
 

It is hard to say what happened, but the I/O hung, and we could not even get to another terminal prompt to examine. There were supposed to be backups for this system, but... Thanks for your answer.

I was presuming you could reboot a Knoppix (or whatever) and redo the rsync. Do a mkfs on the USB, then redo the rsync (maybe) excluding that suspect html file. As I said, make sure it doesn't get interrupted next time.
If you need what's (now) on the USB, that could be way more difficult.

Sync hung under Knoppix
 

The Sync hung under Knoppix, which leads me to believe this is a hardware/chipset issue. There were problems with the handful of chipset manufacturers that implemented native 1.0/1.1. There were device driver - level patches on Windows.

Interesting. Any nfo you'd want to share? (You're invited to email me if you would want to but not in public.)


That doesn't say what situation you're in, which options you've got and which ones you have explored. Speed and error-wise I would prefer booting either KNOPPIX or HELIX (MD5 93a285bfa8ab93d664d508e5b12446d3) and then first cloning the disk(s) (or partitions if the server had a server layout) to a file on a clean, physical disk, external media or another network location (else would taking the disk(s) out and cloning them using another machine be an option?) because having a 'dd' image frees up the machine so you can speed up re-installation and having a 'dd' image could avoid issues you may decide to deal with later on. Tools like Photorec, Testdisk, linen, etc, etc don't need mounted file systems but just /dev partition names or disk image file names avoiding problems due to file system inconsistencies or mount errors. Read-only access to your disk image would also prevent tainting or modifying it beyond salvage, so if you must mount Ext file systems then at least use "-o ro,norecovery,noload" to prohibit replaying and loading journal entries.

More Information
 

Here are more details.

We have booted the PowerEdge with Knoppix.
These problems also originally occurred
when CentOS 5.x was booted and we tried
copying to the stick. After that we
only used Knoppix to boot and perform the copy.

We've formatted with ext3 a 16GB thumb drive.
We've got two of them, and have formatted them
the same way.

The PowerEdge's native USB is probably USB 1.1

Once the system hangs on copy, we've either
yanked the stick or force powered down the system.
Either way, we cannot mount the volume after bringing
the system back up. I can understand why yanking the
stick would be a problem, but not a forced power down.

That's all the information there is.

You might be wondering who would create a DMZ-d server
with no backup. It isn't us. We were called in to rescue
the Mailman distro lists. That's all we care about.

OK. Let's resort to simple questions then.

- How many disks does the server have and what's their size?
- Or, if the server has a server-like partition layout, what is the size of the partition you're trying to get access to?

- Would backing up over the network be an option?
- And if so, if you boot the machine with KNOPPIX or HELIX (don't need a GUI, just multi-user and networking),
- can you get a DHCP lease and
- are you allowed to use their facilities (a machine with an unprivileged account, ample free disk space)?
- Can you 'dd' partitions over the network using netcat or SSH?

- If none of the above, if you boot the machine with any CDROM that has testdisk on it, have testdisk (run it with the "/debug /log" args) access the partition your configuration files are on, can you "explore" the directories you need access too? Are these directories empty or not? Does using the "show deleted files" toggle show deleted files?

More info
 

I appreciate your attention to this question; we're muddling along so far by having throttled down the copying.

To answer your questions:

Backing up over the network isn't really an option.

The server is in a DMZ.

The system has a RAID, and we mount that using Knoppix.

We do get a DHCP lease, and so far, by throttling back
the copy, the sticks have not frozen.

Initially, I was wondering what might be locking up the copy; thought perhaps some attributes had been set on the ASCII Mailman files -- there were not any set -- and perhaps that was locking up the memory stick.

We may never know what causes the lockup, but as I said above, throttling down how much gets copied seems to be working.

Once we get all the data off, we'll wipe the server clean and re-install CentOS. Then, we'll set up rsync; we can ssh into the server. We're also going to set up a mirror server with rsync, so if something goes wrong again, we'll have not only a data backup, but a usable server.

I'd have brought a pc or laptop to use to copy over the network. At usb 1.x you will be there forever.

I'd also run some diags on this. Almost any live cd should be able to be used. If you like Knoppix then fine but the system hanging is odd. The issue could be simply hardware problems.

I have never heard of any proven means that a true only ascii file could compromise a system. An ascii file could in theory contain non-ascii and then could possibly do harm. There are only a few examples ever reported on that and none have been very harmful.

That is one of the reasons why tools like dd are preferred. They don't have any concept of file systems and they don't need to rely on the kernel or user land utilities interpreting a file system's state or the state of its contents nor would they alter the block device or its contents. Sure you need some form of translation to get from a single RAID member to a unified block device you can access but otherwise the problem with basically everything above the basic block device level is that it adds complexity and errors. Using 'dd' or an equivalent means avoiding all of that.