LinuxQuestions.org
Register a domain and help support LQ
Go Back   LinuxQuestions.org > Forums > Other *NIX Forums > Solaris / OpenSolaris
User Name
Password
Solaris / OpenSolaris This forum is for the discussion of Solaris and OpenSolaris.
General Sun, SunOS and Sparc related questions also go here.

Notices



Reply
 
Search this Thread
Old 06-02-2011, 08:46 AM   #1
moata_u
Member
 
Registered: Jan 2011
Posts: 32

Rep: Reputation: 0
Root file system / 100 % full


server root directory is full ..
$df -h
Code:
Filesystem             size   used  avail capacity  Mounted on
/dev/dsk/c1t0d0s0      6.8G   6.7G    16M   100%    /
/devices                 0K     0K     0K     0%    /devices
ctfs                     0K     0K     0K     0%    /system/contract
proc                     0K     0K     0K     0%    /proc
mnttab                   0K     0K     0K     0%    /etc/mnttab
swap                    21G   1.5M    21G     1%    /etc/svc/volatile
objfs                    0K     0K     0K     0%    /system/object
sharefs                  0K     0K     0K     0%    /etc/dfs/sharetab
/platform/SUNW,SPARC-Enterprise-T5120/lib/libc_psr/libc_psr_hwcap2.so.1
                       6.8G   6.7G    16M   100%    /platform/sun4v/lib/libc_psr.so.1
/platform/SUNW,SPARC-Enterprise-T5120/lib/sparcv9/libc_psr/libc_psr_hwcap2.so.1
                       6.8G   6.7G    16M   100%    /platform/sun4v/lib/sparcv9/libc_psr.so.1
fd                       0K     0K     0K     0%    /dev/fd
swap                    21G    10M    21G     1%    /tmp
swap                    21G    40K    21G     1%    /var/run
/dev/dsk/c1t0d0s7      127G    23G   103G    19%    /export/home
**** I tried the following :
-remove the log under /var/log
-file under /var/adm is not that much space in there
-file under /proc directory is taking about 3G space (am not sure if i can remove them) !!
-delete the crash file...


Please any advice , can i increase the size of root ???

** NOTE : that this is database server
 
Old 06-02-2011, 10:31 AM   #2
pingu
Senior Member
 
Registered: Jul 2004
Location: Skuttunge SWEDEN
Distribution: Debian preferably
Posts: 1,318

Rep: Reputation: 124Reputation: 124
Quote:
Originally Posted by moata_u View Post
-file under /proc directory is taking about 3G space (am not sure if i can remove them) !!
That's crazy, shouldn't be any regular files under /proc - also look at output of df saying proc is size 0.

What disks, partitions & space do you have?
You can increase the size of / only if there is free space on the disk next to root-partition. Resizing is normally safe but there is always a risk with this kind of operations. Is this server sensitive to downtime I'd say "no" don't resize.
Another possibility is to use an empty partition and mount it under eg /usr (now I'm talking Linux not sure what you're running - SUN SPARC, is that Solaris then?
If there is no free space left on any disk, then how about swap, do you need 21GB?
Last chance: add another disk and mount some directory there.
 
Old 06-02-2011, 10:45 AM   #3
MensaWater
Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 6,028
Blog Entries: 5

Rep: Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791
Removing log files (or any other files) that are still "open" means that only the name is deleted. The file contents are stored in the inode which won't go away until the process that had the file open stops. I'd try stopping syslogd to see if that helps with the file(s) you arleady deleted. (Also if you're running sendmail it might be writing to /var/log/mail*log. Other processes may be writing to other logs.)

If /var is part of / as it appears to be on your system you should focus on it. /var/log is a good starting point. /var/tmp is another. (In fact here recently we found our Oracle DB apparently had something writing something huge to /var/tmp that caused issues.)
If you do a cd /var then run "du -sk |sort -n" it will show the largest items (files/directories) at the bottom of the output. Check out the largest. (If a directory you can do the same du -sk |sort -n to drill down on its largest.)

Lots of things log or write to /var and it isn't all in /var/log. You don't mention what distro you're on. On RHEL5/CentOS I have often seen /var/cache/yum fill up. (There is a newer yum that fixes this issue.) You can run "yum clean all" to clear that directory. /var/wtmp which stores user logins can get rather large. On Dell PowerEdge Systems with Dell OpenManage installed I've seen large /var/log/TTY* files that can be cleaned but as noted above they are "open" so you have to first shutdown openmanage before clearing them. These are just a few examples.
After you've resolved the issue you might want to consider creating /var as a separate mount (of course you'll need downtime to do that).
 
Old 06-02-2011, 03:51 PM   #4
moata_u
Member
 
Registered: Jan 2011
Posts: 32

Original Poster
Rep: Reputation: 0
first thanks 4 replay ...

am using Sun Solaris 10
i run the command du -sh , in var directory = 70M
and run the command in /proc directory = 374 G "i cant understand !!" all amount is = 150G
 
Old 06-02-2011, 04:32 PM   #5
anomie
Senior Member
 
Registered: Nov 2004
Location: Texas
Distribution: RHEL, Scientific Linux, Debian, Fedora, Lubuntu, FreeBSD
Posts: 3,930
Blog Entries: 5

Rep: Reputation: Disabled
Solaris 10, eh? Let's see output from:
Code:
# find / -xdev -size +94371840c -exec du -h {} \;
 
Old 06-03-2011, 03:05 AM   #6
moata_u
Member
 
Registered: Jan 2011
Posts: 32

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by anomie View Post
solaris 10, eh? Let's see output from:
Code:
# find / -xdev -size +94371840c -exec du -h {} \;
output = nothing
 
Old 06-03-2011, 03:47 AM   #7
moata_u
Member
 
Registered: Jan 2011
Posts: 32

Original Poster
Rep: Reputation: 0
i did restart for the server i have now 2.1G space in root directory !!!

Last edited by moata_u; 06-03-2011 at 05:23 AM.
 
Old 06-03-2011, 05:43 AM   #8
jschiwal
Guru
 
Registered: Aug 2001
Location: Fargo, ND
Distribution: SuSE AMD64
Posts: 15,733

Rep: Reputation: 655Reputation: 655Reputation: 655Reputation: 655Reputation: 655Reputation: 655
The /proc directory is a pseudo filesystem. The files are created by the kernel when you read them. It isn't taking up GBs of space.

One thing to look at is deleted files which are still open. Restarting a service may clear that up. This is done to prevent name collision, and prevent a stale file from remaining in the event the service crashed.

Another user had a problem with a new and an old instance of a java service running. Each had a deleted file open around 10GB of size.

You can detect open deleted files with: lsof +L1

Another thing could be files in /tmp and /var/tmp taking up a lot of space. Your server may be configured to delete files in /tmp and /var/tmp during a reboot.

---
The root partition is very small to begin with. Your /tmp and /var/tmp are almost 3 times as large.

I don't understand this lines:
Quote:
SUNW,SPARC-Enterprise-T5120/lib/libc_psr/libc_psr_hwcap2.so.1
6.8G 6.7G 16M 100% /platform/sun4v/lib/libc_psr.so.1
/platform/SUNW,SPARC-Enterprise-T5120/lib/sparcv9/libc_psr/libc_psr_hwcap2.so.1
6.8G 6.7G 16M 100% /platform/sun4v/lib/sparcv9/libc_psr.so.1
Why are the devices and mount points library files?
What is the purpose of /export/home? It looks like you are using a database server as a file server for users HOME directories. Is this correct?
 
Old 06-03-2011, 10:07 AM   #9
MensaWater
Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 6,028
Blog Entries: 5

Rep: Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791
Quote:
Originally Posted by moata_u View Post
i did restart for the server i have now 2.1G space in root directory !!!
As I noted in my earlier post when you delete files that are "open" only the names are deleted. The actual space of files and the inodes are NOT deleted. Stopping the processes that have the files open will close the files and allow the deletion to complete.

If you reboot the server as you did then you have stopped ALL processes including those that had files open and thereby resolved the issue.

The moral of the story: Do NOT delete log files or other files that are written to frequently - it is better to do ">file" which will truncate the file to 0 bytes.

Also you can search for processes that have files open with no name by using the lsof command and looking for large CHR files in its output that have no name. The PID will be the process that has the file open and killing the process will free the space.
 
Old 06-03-2011, 10:39 AM   #10
moata_u
Member
 
Registered: Jan 2011
Posts: 32

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by jschiwal View Post
The /proc directory is a pseudo filesystem. The files are created by the kernel when you read them. It isn't taking up GBs of space.

One thing to look at is deleted files which are still open. Restarting a service may clear that up. This is done to prevent name collision, and prevent a stale file from remaining in the event the service crashed.

Another user had a problem with a new and an old instance of a java service running. Each had a deleted file open around 10GB of size.

You can detect open deleted files with: lsof +L1

Another thing could be files in /tmp and /var/tmp taking up a lot of space. Your server may be configured to delete files in /tmp and /var/tmp during a reboot.

---
The root partition is very small to begin with. Your /tmp and /var/tmp are almost 3 times as large.

I don't understand this lines:

Why are the devices and mount points library files?
What is the purpose of /export/home? It looks like you are using a database server as a file server for users HOME directories. Is this correct?


yes , you are right am installing database server , so what is your advice ?? this server contain primary database for Billing system , and this company for WIMAX services still not lunch !! should i format the server and re partition correctly ???
 
Old 06-03-2011, 10:41 AM   #11
moata_u
Member
 
Registered: Jan 2011
Posts: 32

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by MensaWater View Post
As I noted in my earlier post when you delete files that are "open" only the names are deleted. The actual space of files and the inodes are NOT deleted. Stopping the processes that have the files open will close the files and allow the deletion to complete.

If you reboot the server as you did then you have stopped ALL processes including those that had files open and thereby resolved the issue.

The moral of the story: Do NOT delete log files or other files that are written to frequently - it is better to do ">file" which will truncate the file to 0 bytes.

Also you can search for processes that have files open with no name by using the lsof command and looking for large CHR files in its output that have no name. The PID will be the process that has the file open and killing the process will free the space.
this server is the primary database for billing system , and its not accepted that every period of time that i need to restart the server ....Please your advice ...
 
Old 06-03-2011, 12:15 PM   #12
MensaWater
Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 6,028
Blog Entries: 5

Rep: Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791
My advice was in the post - see the use of lsof I referenced.

You should not need to reboot - I was merely explaining why the reboot helped.

Prior to deleting a file you can run lsof against the file itself and it will show you if anything is using it.
 
Old 06-03-2011, 09:04 PM   #13
ComputerErik
Member
 
Registered: Apr 2005
Location: NYC
Distribution: Debian, RHEL
Posts: 268

Rep: Reputation: 42
You asked if you should start again from scratch, but also that this is a critical database server. What is the case, is this a production system which can have no downtime, or is it a new setup which you can do over?

It does seem like the partition scheme chosen is rather poor for your intentions. Where will the database files go? If doing a fresh install and choosing a sensible partition scheme is an option I would definitely recommend that route if this is a new system being setup to go in a critical role. If that is not an option then take the previous advice and look at shrinking one of the other partitions so you can grow /.
 
Old 06-05-2011, 04:50 AM   #14
jschiwal
Guru
 
Registered: Aug 2001
Location: Fargo, ND
Distribution: SuSE AMD64
Posts: 15,733

Rep: Reputation: 655Reputation: 655Reputation: 655Reputation: 655Reputation: 655Reputation: 655
If this is a new install, I would advice a fresh installation with the bulk of the space taken for where the database is contained. I imagine this would need much more space when it goes online or has been used for a while.

For example, for a mysql database, the /var/ directory would probably contain the database. Some people create a separate partition for both /var and /var/log/ on servers. The later is to prevent an attacker from filling up the logs and depriving the root partition of disk space. On my distribution, mysql uses /var/lib/ while other services such as apached tend to use /srv/. It may depend on which database you are using and even the version.

If this is a server, does it use raid for redundancy?

Your post is in Linux Server but it seems to be for a Solaris machine. There will be some difference for Solaris such as the partition names, but many of the principles are the same. On a Linux server, world writable directories should be mounted on a partition that used the noexec and nodev mount options. If you have these same mount options, you may want the partition with /tmp to be mounted this way as well.

I still don't understand the partitions directories with the names of libraries. Did you do that manually? Or is that a Solaris thing. It almost looks hackey to my eyes.

Last edited by jschiwal; 06-05-2011 at 04:52 AM.
 
Old 06-05-2011, 06:44 AM   #15
jlliagre
Moderator
 
Registered: Feb 2004
Location: Outside Paris
Distribution: Solaris10, Solaris 11, Mint, OL
Posts: 9,523

Rep: Reputation: 365Reputation: 365Reputation: 365Reputation: 365
Quote:
Originally Posted by jschiwal View Post
I still don't understand the partitions directories with the names of libraries. Did you do that manually? Or is that a Solaris thing. It almost looks hackey to my eyes.
This is standard with Solaris. If a hardware platform specific enhanced libc exists, it is mounted on top of the default one.
http://src.opensolaris.org/source/xr...one/fs-root#29

Last edited by jlliagre; 06-05-2011 at 06:45 AM.
 
  


Reply

Tags
admin, server, space


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] System File is Full (100%) How can I relieve the situation? mike-member Linux - Newbie 8 03-14-2011 03:32 AM
root file system full linuxtesting2 Solaris / OpenSolaris 7 08-15-2007 06:41 AM
root file system full WLReed Ubuntu 2 03-28-2007 12:29 AM
Root file system shows 100% percent utilization simplyrahul Linux - Software 1 11-26-2004 11:53 AM
/home file system 100% full gromo Linux - Enterprise 5 11-04-2004 10:47 PM


All times are GMT -5. The time now is 11:30 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration