Root file system / 100 % full

moata_u · 06-02-2011, 07:46 AM

server root directory is full ..
$df -h

Code:

Filesystem             size   used  avail capacity  Mounted on
/dev/dsk/c1t0d0s0      6.8G   6.7G    16M   100%    /
/devices                 0K     0K     0K     0%    /devices
ctfs                     0K     0K     0K     0%    /system/contract
proc                     0K     0K     0K     0%    /proc
mnttab                   0K     0K     0K     0%    /etc/mnttab
swap                    21G   1.5M    21G     1%    /etc/svc/volatile
objfs                    0K     0K     0K     0%    /system/object
sharefs                  0K     0K     0K     0%    /etc/dfs/sharetab
/platform/SUNW,SPARC-Enterprise-T5120/lib/libc_psr/libc_psr_hwcap2.so.1
                       6.8G   6.7G    16M   100%    /platform/sun4v/lib/libc_psr.so.1
/platform/SUNW,SPARC-Enterprise-T5120/lib/sparcv9/libc_psr/libc_psr_hwcap2.so.1
                       6.8G   6.7G    16M   100%    /platform/sun4v/lib/sparcv9/libc_psr.so.1
fd                       0K     0K     0K     0%    /dev/fd
swap                    21G    10M    21G     1%    /tmp
swap                    21G    40K    21G     1%    /var/run
/dev/dsk/c1t0d0s7      127G    23G   103G    19%    /export/home

**** I tried the following :
-remove the log under /var/log
-file under /var/adm is not that much space in there
-file under /proc directory is taking about 3G space (am not sure if i can remove them) !!
-delete the crash file...

Please any advice , can i increase the size of root ???

** NOTE : that this is database server

pingu · 06-02-2011, 09:31 AM

Quote:

Originally Posted by moata_u

-file under /proc directory is taking about 3G space (am not sure if i can remove them) !!

That's crazy, shouldn't be any regular files under /proc - also look at output of df saying proc is size 0.

What disks, partitions & space do you have?
You can increase the size of / only if there is free space on the disk next to root-partition. Resizing is normally safe but there is always a risk with this kind of operations. Is this server sensitive to downtime I'd say "no" don't resize.
Another possibility is to use an empty partition and mount it under eg /usr (now I'm talking Linux not sure what you're running - SUN SPARC, is that Solaris then?
If there is no free space left on any disk, then how about swap, do you need 21GB?
Last chance: add another disk and mount some directory there.

MensaWater · 06-02-2011, 09:45 AM

Removing log files (or any other files) that are still "open" means that only the name is deleted. The file contents are stored in the inode which won't go away until the process that had the file open stops. I'd try stopping syslogd to see if that helps with the file(s) you arleady deleted. (Also if you're running sendmail it might be writing to /var/log/mail*log. Other processes may be writing to other logs.)

If /var is part of / as it appears to be on your system you should focus on it. /var/log is a good starting point. /var/tmp is another. (In fact here recently we found our Oracle DB apparently had something writing something huge to /var/tmp that caused issues.)
If you do a cd /var then run "du -sk |sort -n" it will show the largest items (files/directories) at the bottom of the output. Check out the largest. (If a directory you can do the same du -sk |sort -n to drill down on its largest.)

Lots of things log or write to /var and it isn't all in /var/log. You don't mention what distro you're on. On RHEL5/CentOS I have often seen /var/cache/yum fill up. (There is a newer yum that fixes this issue.) You can run "yum clean all" to clear that directory. /var/wtmp which stores user logins can get rather large. On Dell PowerEdge Systems with Dell OpenManage installed I've seen large /var/log/TTY* files that can be cleaned but as noted above they are "open" so you have to first shutdown openmanage before clearing them. These are just a few examples.
After you've resolved the issue you might want to consider creating /var as a separate mount (of course you'll need downtime to do that).

moata_u · 06-02-2011, 02:51 PM

first thanks 4 replay ...

am using Sun Solaris 10
i run the command du -sh , in var directory = 70M
and run the command in /proc directory = 374 G "i cant understand !!" all amount is = 150G

anomie · 06-02-2011, 03:32 PM

Solaris 10, eh? Let's see output from:

Code:

# find / -xdev -size +94371840c -exec du -h {} \;

moata_u · 06-03-2011, 02:05 AM

Quote:

Originally Posted by anomie

solaris 10, eh? Let's see output from:

Code:

# find / -xdev -size +94371840c -exec du -h {} \;

output = nothing

moata_u · 06-03-2011, 02:47 AM

i did restart for the server i have now 2.1G space in root directory !!!

jschiwal · 06-03-2011, 04:43 AM

The /proc directory is a pseudo filesystem. The files are created by the kernel when you read them. It isn't taking up GBs of space.

One thing to look at is deleted files which are still open. Restarting a service may clear that up. This is done to prevent name collision, and prevent a stale file from remaining in the event the service crashed.

Another user had a problem with a new and an old instance of a java service running. Each had a deleted file open around 10GB of size.

You can detect open deleted files with: lsof +L1

Another thing could be files in /tmp and /var/tmp taking up a lot of space. Your server may be configured to delete files in /tmp and /var/tmp during a reboot.

---
The root partition is very small to begin with. Your /tmp and /var/tmp are almost 3 times as large.

I don't understand this lines:

Quote:

SUNW,SPARC-Enterprise-T5120/lib/libc_psr/libc_psr_hwcap2.so.1
6.8G 6.7G 16M 100% /platform/sun4v/lib/libc_psr.so.1
/platform/SUNW,SPARC-Enterprise-T5120/lib/sparcv9/libc_psr/libc_psr_hwcap2.so.1
6.8G 6.7G 16M 100% /platform/sun4v/lib/sparcv9/libc_psr.so.1

Why are the devices and mount points library files?
What is the purpose of /export/home? It looks like you are using a database server as a file server for users HOME directories. Is this correct?

MensaWater · 06-03-2011, 09:07 AM

Quote:

Originally Posted by moata_u

i did restart for the server i have now 2.1G space in root directory !!!

As I noted in my earlier post when you delete files that are "open" only the names are deleted. The actual space of files and the inodes are NOT deleted. Stopping the processes that have the files open will close the files and allow the deletion to complete.

If you reboot the server as you did then you have stopped ALL processes including those that had files open and thereby resolved the issue.

The moral of the story: Do NOT delete log files or other files that are written to frequently - it is better to do ">file" which will truncate the file to 0 bytes.

Also you can search for processes that have files open with no name by using the lsof command and looking for large CHR files in its output that have no name. The PID will be the process that has the file open and killing the process will free the space.

moata_u · 06-03-2011, 09:39 AM

Quote:

Originally Posted by jschiwal

The /proc directory is a pseudo filesystem. The files are created by the kernel when you read them. It isn't taking up GBs of space.

One thing to look at is deleted files which are still open. Restarting a service may clear that up. This is done to prevent name collision, and prevent a stale file from remaining in the event the service crashed.

Another user had a problem with a new and an old instance of a java service running. Each had a deleted file open around 10GB of size.

You can detect open deleted files with: lsof +L1

Another thing could be files in /tmp and /var/tmp taking up a lot of space. Your server may be configured to delete files in /tmp and /var/tmp during a reboot.

---
The root partition is very small to begin with. Your /tmp and /var/tmp are almost 3 times as large.

I don't understand this lines:

Why are the devices and mount points library files?
What is the purpose of /export/home? It looks like you are using a database server as a file server for users HOME directories. Is this correct?

yes , you are right am installing database server , so what is your advice ?? this server contain primary database for Billing system , and this company for WIMAX services still not lunch !! should i format the server and re partition correctly ???

moata_u · 06-03-2011, 09:41 AM

Quote:

Originally Posted by MensaWater

As I noted in my earlier post when you delete files that are "open" only the names are deleted. The actual space of files and the inodes are NOT deleted. Stopping the processes that have the files open will close the files and allow the deletion to complete.

If you reboot the server as you did then you have stopped ALL processes including those that had files open and thereby resolved the issue.

The moral of the story: Do NOT delete log files or other files that are written to frequently - it is better to do ">file" which will truncate the file to 0 bytes.

Also you can search for processes that have files open with no name by using the lsof command and looking for large CHR files in its output that have no name. The PID will be the process that has the file open and killing the process will free the space.

this server is the primary database for billing system , and its not accepted that every period of time that i need to restart the server ....Please your advice ...

MensaWater · 06-03-2011, 11:15 AM

My advice was in the post - see the use of lsof I referenced.

You should not need to reboot - I was merely explaining why the reboot helped.

Prior to deleting a file you can run lsof against the file itself and it will show you if anything is using it.

ComputerErik · 06-03-2011, 08:04 PM

You asked if you should start again from scratch, but also that this is a critical database server. What is the case, is this a production system which can have no downtime, or is it a new setup which you can do over?

It does seem like the partition scheme chosen is rather poor for your intentions. Where will the database files go? If doing a fresh install and choosing a sensible partition scheme is an option I would definitely recommend that route if this is a new system being setup to go in a critical role. If that is not an option then take the previous advice and look at shrinking one of the other partitions so you can grow /.

jschiwal · 06-05-2011, 03:50 AM

If this is a new install, I would advice a fresh installation with the bulk of the space taken for where the database is contained. I imagine this would need much more space when it goes online or has been used for a while.

For example, for a mysql database, the /var/ directory would probably contain the database. Some people create a separate partition for both /var and /var/log/ on servers. The later is to prevent an attacker from filling up the logs and depriving the root partition of disk space. On my distribution, mysql uses /var/lib/ while other services such as apached tend to use /srv/. It may depend on which database you are using and even the version.

If this is a server, does it use raid for redundancy?

Your post is in Linux Server but it seems to be for a Solaris machine. There will be some difference for Solaris such as the partition names, but many of the principles are the same. On a Linux server, world writable directories should be mounted on a partition that used the noexec and nodev mount options. If you have these same mount options, you may want the partition with /tmp to be mounted this way as well.

I still don't understand the partitions directories with the names of libraries. Did you do that manually? Or is that a Solaris thing. It almost looks hackey to my eyes.

jlliagre · 06-05-2011, 05:44 AM

Quote:

Originally Posted by jschiwal

I still don't understand the partitions directories with the names of libraries. Did you do that manually? Or is that a Solaris thing. It almost looks hackey to my eyes.

This is standard with Solaris. If a hardware platform specific enhanced libc exists, it is mounted on top of the default one.
http://src.opensolaris.org/source/xr...one/fs-root#29