Input/output error

rob19 · 08-25-2004, 06:19 AM

I have been running a Web server with RH9 for about six months now. Everything has been going fine and I have not made any changes for a while. However, I have just noticed that Webalizer is no longer producing reports properly (they seem garbled) and now I cannot even copy files via FTP or even do a mkdir without getting an error (see below).

Code:

mkdir testing
mkdir: cannot create directory `testing': Input/output error

I tried to search for this and found people had similar errors but they mostly seemed to be tampering with their partitions and I have not been doing this. One previous entry mentioned the fix being...

Code:

mke2fs -c  -b

From what I can tell this will create an ext2/3 filesystem. Will this format or wipe any of my existing data? This is a production Web server and I do not want to have to rebuild everything or lose any data.

Many thanks in advance,

chrism01 · 08-25-2004, 07:18 AM

I wouldn't make an FS...
Sounds like it could be a superblock/disk corruption in that partition. Can you create a simple file there? Also try running df, du and see if you get expected results.
Check /var/log/messages at the same time as trying the file and dir creates; ie open a separate xterm and run
tail -f /var/log/messages
and watch for any reports.
Is your logrotate working? Check the contents of /var/log dir.
Ultimately (after a full backup), try a machine reboot and see if it does a fsck.
HTH

rob19 · 08-25-2004, 04:11 PM

df seems to work but du doesn't. Also I can't restart the server.

Code:

[root@tiger root]# du
-bash: /usr/bin/du: Input/output error
[root@tiger root]# df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/hda2            114247528   2110896 106333152   2% /
/dev/hda1               101089      9294     86576  10% /boot
none                    256924         0    256924   0% /dev/shm
[root@tiger root]# shutdown -r now
-bash: /sbin/shutdown: Input/output error

chrism01 · 08-25-2004, 04:22 PM

This guy had the same prob and it was a bad disk block: http://www.webhostingtalk.com/archiv.../190656-1.html
If you check the /var/log/messages and /var/log/dmesg you may find warnings re your disk.
You're going to have to shutdown the apps if possible, and try backing up what you can (if you haven't already), then either try "fsck" from the cmd line (as root), or try "init 6" which should force a reboot.
Maybe trying the backup remotely will work (or not...)

Worst case, shutdown every service you can and pull the plug.... good luck

rob19 · 08-25-2004, 05:10 PM

I tried the fsck command from a remote shell as root. I wasn't sure if it was safe to cause SEVERE filesystem damage or if this command should only be run when the server is starting up. Advice?

Code:

[root@tiger root]# fsck -A -V -a
fsck 1.32 (09-Nov-2002)
Checking all file systems.
[/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/hda2
/dev/hda2 is mounted.

WARNING!!!  Running e2fsck on a mounted filesystem may cause
SEVERE filesystem damage.

Do you really want to continue (y/n)? no

check aborted.
[/sbin/fsck.ext3 (1) -- /boot] fsck.ext3 -a /dev/hda1
/dev/hda1 is mounted.  e2fsck: Cannot continue, aborting.

rob19 · 08-25-2004, 05:22 PM

I did a cat on /var/log/dmesg like you said but did not see anything that struck me as obviously wrong. I noticed one line that mentioned adding swap though...

Code:

EXT3 FS 2.4-0.9.19, 19 August 2002 on ide0(3,2), internal journal
Adding Swap: 1044216k swap-space (priority -1)
kjournald starting.  Commit interval 5 seconds
EXT3 FS 2.4-0.9.19, 19 August 2002 on ide0(3,1), internal journal
EXT3-fs: mounted filesystem with ordered data mode.

rob19 · 08-25-2004, 05:42 PM

That other guys problem they didn't mention how they fixed it but he seemed pretty happy at the end. They did mention fsck again so I thought I might as well go ahead and try it. Looks like I have a bad block (possibly more). I am going to try using badblocks command to see if that will repair it.

Code:

[root@tiger root]# fsck -A -V -a
fsck 1.32 (09-Nov-2002)
Checking all file systems.
[/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/hda2
/dev/hda2 is mounted.

WARNING!!!  Running e2fsck on a mounted filesystem may cause
SEVERE filesystem damage.

Do you really want to continue (y/n)? yes

/: recovering journal
/: Attempt to read block from filesystem resulted in short read while reading block 5443

JBD: Failed to read block at offset 4910
fsck.ext3: Input/output error while recovering ext3 journal of /

rob19 · 08-25-2004, 05:48 PM

Lousy thing!

Code:

[root@tiger root]# badblocks -n /dev/hda1
-bash: /sbin/badblocks: Input/output error

Looks like I am going to have to drive into the city, go to the ISP and pull the plug on the server.

rob19 · 08-26-2004, 03:38 AM

Well restarting didn't fix it. But pulling the power plug out completely did. I'm not kidding. There must have been something funky going on with the hard drive but with the power cut completely it has been behaving fine ever since and fsck finds nothing wrong - all clean.

chrism01 · 08-26-2004, 06:27 AM

Certainly sounds odd. I'd do a full backup if I was you

Given that moving objects (disks) pull the most power, I suppose it's just possible that the power connection was slightly loose/out of position.
BTW, dmesg is more of a startup log I think. /var/log/messages is the on-going log.
Keep a strict eye on it for a week or so and see what happens...

jhermans · 02-15-2007, 05:32 PM

This is the second time I've had this problem. Anyways, thought I could throw in some good information I picked up on the web, this time around. You can reboot even in this situation remotely (I did it over SSH) by doing a 'hard reboot.'

Forced Reboot

Code:

echo 1 > /proc/sys/kernel/sysrq
echo b > /proc/sysrq-trigger

Forced Shutdown

Code:

echo 1 > /proc/sys/kernel/sysrq
echo o > /proc/sysrq-trigger

Courtesy of http://blog.air4web.com/linux-force-reboot.html

sureshtheory · 12-23-2010, 12:48 AM

Hi,

Still last day when i try to open /var/log/messages working fine with SLES10 sp3... after booting with some other server and i took back to native server.Itz bootin fine.. but when i try to open /var/log/messages .. it is giving me :/var/log/messages : Input/output error ..

Anyone has any clue on this????

sureshtheory · 12-23-2010, 01:18 AM

Hi,

/var/log/messages working fine.. i just deleted /var/log/messages and created one file as messages and rebooted..... itz workin now..

my first post replied by me itsel...

lneto77 · 02-28-2012, 10:23 AM

@jhermans

Fantastic tip man, i was stuck on the same situation and I've managed to get my server rebooted.

Thanks a lot.

Spechal · 08-21-2012, 09:35 PM

Same issue on CentOS 6. I was able to reboot with init 6.