Hi everyone, i saw something similar to this being posted years ago but nobody provided any solution as that person merely restarted his server and things went back to normal. It doesn't happen to mine though. Below is the output of df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda5 7.7G 465M 6.9G 7% /
/dev/sda7 243M 5.0M 225M 3% /boot
/dev/sda1 100M 5.5M 95M 6% /boot/efi
/dev/sda8 20G 17G 2.6G 87% /home
/dev/sdd2 4.9G 3.4G 1.3G 74% /home2
/dev/sdc1 68G 62G 2.2G 97% /op2
/dev/sdb1 68G 65G 0 100% /opt
/dev/sdd1 63G 58G 2.0G 97% /opt3
none 32G 0 32G 0% /dev/shm
/dev/sda2 9.7G 8.8G 403M 96% /swap
/dev/sda9 5.8G 33M 5.5G 1% /swap2
/dev/sda6 4.9G 33M 4.6G 1% /tmp
/dev/sda4 9.7G 5.7G 3.5G 63% /usr
/dev/sda3 9.7G 1.7G 7.5G 19% /var
/dev/sdc1 68G 62G 2.2G 97% /mnt/usbdisk1
/dev/sdg 459G 401G 35G 92% /mnt/usbdisk2
/dev/sdf1 276G 215G 47G 83% /mnt/usbdisk3

As you can see, for filesystem /dev/sdb1, even though there is 3G free space, the % used is still 100%. This will not serve as any problem if our daily activities aren't affected. Sadly, only root is able to create new files in that folder.

drwxr-xr-x 6 root root 4096 Aug 13 2004 .
drwxr-xr-x 29 root root 4096 Feb 26 10:24 ..
drwxr-x--- 5 gtsyna gtsyna 4096 Feb 7 2007 blast
drwxr-xr-x 6 root root 4096 Sep 27 2004 intel
drwx------ 2 root root 16384 Aug 13 2005 lost+found
drwxrwxr-x 30 gtsyna gtsyna 4096 Feb 26 10:30 synamatix

Even though i logged in as gtsyna and go into synamatix to create new files, i will get this error : E12: Error write in swap file.

[root@mgrc-t4svr-01 ~]# swapon -s
Filename Type Size Used Priority
/swap/swapfile file 9113568 0 -1

I noticed that i am using only swap files to provide swap space ( created by previous linux admin) and strangely, it doesn't utilize the other swap filesystem as listed in the df output. I have no idea why. Any recommendations and advices? I have run fsck on /opt by unmounting and running it and result came out clean. I even rebooted the server but the problem persists. Should i just destroy the swap filesystems and utilize swap space instead of swap file? PLease advise. Thank you.

When a file system is created, a percentage is reserved for super-user use to allow for system administration activities even if the disk fills up. That's probably what is happening with /dev/sdb1.

Have you checked whether there are unused applications (or sources) under /opt that can be removed?

My personal preference is to use a swap partition rather than create a swap file on a partition. I don't have any data available here on which is more efficient. The best solution is to get more RAM if you can afford it.

Hi Gilead, it was previously 100% utilized, and a real 68G utilized but the application team has deleted loads of files, hence the reduction of 3G from previous utilization value. But, it doesn't show in the %use column. Swap space is not an issue because swap is lowly used :
total used free shared buffers cached
Mem: 65170 1574 63595 0 109 1082
-/+ buffers/cache: 383 64786
Swap: 8899 0 8899

My concern is just that whether my swap file is corrupted or it cannot detect the empty space in /opt because i am not allowed to create any new file even though there is 3G free space. What can i do?

There are 2 things happening there. Firstly, /dev/sdb1 (/opt) filled up and secondly /dev/sda2 (/swap) filled up.

For the first one, since you've already re-booted the box and run fsck on it, I don't think the problem is related to deleted files that have not been released. There's some info in the SLUG mailing list archives about this. By default there is 5% (about 3.4GB in this case) of the operating system reserved for the super-user. Can you remove, even temporarily, a few more gigs of data and see if normal users can write to the file system again?

For the second one, I think your swap file is filling up the /dev/sda2 partition and the "E12: Error write in swap file" error occurs when it runs out of space. If you remove the swap file, change the partition to type swap, then re-create it as a swap partition then this should not happen.

Hi Gilead, guess what i did. I destroyed the 2 swap filesystems and created a swap space with 2G instead but i didn't touch the /opt filesystem as i cannot afford to format it due to the large amount of important data sitting on it. Even after i have restructured, the /opt is still showing 100% use, normal users can never create files in that /opt but root can. Hence i deduce that it has nothign to do with the swap anymore but more on the /opt or something else that controls the free space of filesystem.

That still leaves the suggestion from earlier though - can you move files from the /opt partition so that more that 3.4GB is available? That would rule out whether the problem is due to the partition being too full and only having the default 5% available for the super-user.