Dell Poweredge 2950 with CentOS 5.2 acting all funky this morning.
 

The usual Monday morning hot steamy cup of Joe.

So I have a Dell PowerEdge 2950 here running Asterisk for our phone system. This morning, after a great weekend where I got to catch up on my reading and enjoy some nice time with the girlfriend, one of my users reports he is unable to delete a message on his phone. I verify the problem and find out that I can't either. IN fact here is the error I got when I attempted.

So at first I am thinking that Asterisk is screwy and needs a good flogging. But after looking at the errors with the file system becoming read only I begin to wonder. Has my precious file system become corrupt?

I attempt to make a back-up with my weekend changes; however, I can't run scp or ssh to move stuff to the other server. I was getting a bus error when I tried ssh or scp.

The server became un-responsive while trying to mount a USB drive and the phones "go orange". Last resort I reboot the server, it comes online with no errors and things appear to be working appropriately.

What checks should I run? fsck for sure, but is there anything else I can look at memory wise that would have caused the bus error? Any thoughts? Things are acting normal now, maybe it was a fluke, but maybe something is seriously wrong.

/me hates computers.

Is it /var a separate fs? Check if your fs for some reason didnt remount itself due to some failure in read only mode.

This can happen to / root fs is some cases and if u dont have a separate fs for it, it might have happened.

Sorry didn't look at the post properly... what I meant to say is that do you have a proper settings on the system? Meaning did you set up multiple fs's for your system or only / root fs?.

To avoid something like this you should try to create or do something like this:

Build based on LVM and we gonna call it rootvg:

first you have /boot which cant be included in the VG. This fs can have like max 100MB, don't need more then that.

The root / fs which should be in rootvg with lets say 1GB, you can increase this as you go, on the fly.

The you need this /usr /home /var /var/log /opt and maybe /var/spool as it might be in used often.

All of this in the rootvg as Logical Volumes. Of course you also need a Logical Volume for swap which you could call swap_lv, but it doesn't matter and the size's of this fs's should be as much as you need. You don't have to add the full size or max that you have on the disk's, because with the LVM and ext2online you could resize/extend/reduce the LV's on the fly without any issue.

There also is a settings by default for "/" root fs in /etc/fstab something like remount-ro which means that the kernel will automatically remount the root / fs in read-only mode in case of some problems like I/O, hardware, or problems with the fs like ... inode issues and so on.

Anway for this kind of issue, you should watch the logs, dmesg or setup in syslog.conf something like

*.debug /var/log/syslog.out (for everything to be logged in syslog.out)

Also good idea to set the systems in RAID (hardware better) but software could work too, or you could do mirrored VG.

Good luck!