linux server hangs
 

I have 2 redhat linux 8.0 running on p4, 40gb hdd, 512mb RAM.
it works fine but sometimes suddenly it hangs for 20-30 minutes then i have to reboot the machine manually. when it hangs it won't allow me to do anything also users who have logged in from other machines via telnet can't do anything. our machine is running all the time 24/7. I checked /var/log/messages and did not find any errors. what could be the problem and also how do i investigate what problem like OS prob or anyother prob.

Thanks in advance

Would you be running updatedb at too high a priority? What time is it scheduled and would that correspond to when it happens? Sometimes drive activity can chug a system if run at to high a priority.

man nice

I don't have any DB installed in it and do i don't do any dbupdate, just i have apache(intranet) , samba installed and around 10 users using the machine for general purpose like writing few shell scrips..

You do have updatedb installed if you can use the locate command.

What happens if you type something like:

locate updatedb

If you do not get a command not found error, you have it installed and there is a cron job that runs it once a day early am in most distros.

after running locate updatedb i got this . i doubt some services giving problem... i have apache, samba, running on the machine along with that we have installed latest perl, rrdtool etc.. so how do i overcome this linux hanging problem. it is not hanging all the time, once or twice a day for 20 mins to 30 mins.

/usr/bin/updatedb
/usr/share/man/man1/updatedb.1.gz
/usr/share/man/ja/man1/updatedb.1.gz
/usr/share/man/it/man1/updatedb.1.gz
/etc/updatedb.conf

You have to trace down the cause of the hang by reading appropriate logs and looking for some sort of pattern that it may follow

Does the system respond again after the 20-30 minutes? If so, then something is chugging down the system, but the system hasn't crashed, it's busy doing something that isn't causing error messages to be logged

Can you tie it to a specific time period? Could it happen only once a day? Always twice a day.

Look for any patterns of behavior.

well thanks for your quick response. no there is no specific time it hangs, some time once a day it hangs and some time after 2 days or 3 days. and as i told u i don't have much applications or s/w installed in it. i checked /var/log/messages for errors it didn't show any err messages. sometime it was getting stuck when starting smclient and sendmail service at boot time and we don't hv any mail server in and. i have stopped both the services and also disabled X (init 5 from /etc/inittab) and now its directly going to init 3. so any other things i can chk to make sure whether any services giving prob or h/w is giving prob or any bug... thanks for your help.

Hi one more thing, yes after hanging it takes 15 mins to 30 mins then it comes back to normal, sometimes only one user will be using or nobody is using then it comes back to normal position. there is no specific time any time it can hang.

Servers should have a separate /var partition, for one thing. Lets investigate your system.

What is your partitioning scheme?
# How is the hard drive partitioned
fdisk /dev/hd<X> -l

Memory usage?
# Memory and swap information
free

Disk room free?
# How much free drive space
df -h

Check /var
# Show disk usage by current directory and all subdirectories
du | less

Is there anything suspicious?

Hi,
/var has taken 97 % space, how do i add some more space to it i mean free space to it?? and plz find my system report and advice.

this is my system report :-

free :-

total used free shared buffers cached
Mem: 246612 52612 194000 0 20528 21192
-/+ buffers/cache: 10892 235720
Swap: 2112536 0 2112536

df -h
Filesystem Size Used Avail Use% Mounted on
/dev/hda2 7.9G 240M 7.2G 4% /
/dev/hda1 243M 17M 213M 8% /boot
/dev/hda8 9.3G 987M 7.8G 11% /home
/dev/hda6 4.8G 33M 4.5G 1% /opt
none 120M 0 120M 0% /dev/shm
/dev/hda7 5.9G 4.0G 1.5G 72% /usr
/dev/hda5 5.9G 5.4G 188M 97% /var

That's your problem, alright, you haven't been cleaning out your logs after reviewing them like a good sysadmin, eh :p

Look in /var/log and delete your rotated logs (generally named <something>.<some number> to start.

Analyze what the server does and keep an eye on var logs as they can grow considerably.

BTW, I have no experience running a server, just stuff I have found reading stuff like Proper Filesystem Layout

Hi In my /var there are so many dir's in that can i delete files from mqueue and clientmqueue it has acquired so much space.... plz advice.
thanks

hi
heheheh :)) i have not checked log files from few months
now I have created some space in /var/log and now the /var capacity is 53 %. hope it will not hang for some more time :)) and advice for any other tuning or checking..

thanks

You made your / way to large, however (almost 8 gig and you are using 240 MB). You don't need a very big one at all with a /usr partition.

For the easiest solution to get more /var room, I would suggest adding another drive with a /var/log partition large enough to hold the logs you need as you don't have much room left in /var as you have the system set up.

Could you make a complete back up and re-do the partitioning better? That would be the most efficient use of your existing disk space. I am on pain medication just now and find it hard to think very well just now. Good night.

Good, you got a start and it's running and with some looking after, it will run OK and you can think and plan the layout. I will paste some stuff I am in the process of writing up on moving stuff around and using symbolic links,. Maybe that will give you some ideas.

# Clone a distro to another drive or move directories around
Install drive in box (assuming IDE1 slave for examples) and ensure BIOS can detect it. Boot into Linux and login to your user's account. Open an x terminal and partition and format the new drive as you wishExit with w to write the partition table.

Format the partitions with the chosen filesystems:
mke2fs /dev/hdbX -> ext2
mke2fs -j /dev/hdbX -> ext3
mkswap -> swap
mkreiserfs -> reiserfs
mkfs.xfs -> xfs

Make directories for source and destination mount points. You need these to keep out of an endless loop of copying itsself over and over.Mount your os partitions that you want to clone on /mnt/source
Mount your partitions on the new drive, making directory entries for your separate partitions (/boot, /home, /var etc.). If you are just copying a directory such as var, just cd to the directory you wish to copy instead of mounting partitions on /mnt/source.

Now, cd to /mnt/source and pipe it over with tarEdit /etc/fstab as needed, install the boot loader if you moved a whole distro, and it should work when it is installed in it's final position.