LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Other *NIX Forums > Solaris / OpenSolaris
User Name
Password
Solaris / OpenSolaris This forum is for the discussion of Solaris, OpenSolaris, OpenIndiana, and illumos.
General Sun, SunOS and Sparc related questions also go here. Any Solaris fork or distribution is welcome.

Notices


Reply
  Search this Thread
Old 10-15-2009, 08:24 AM   #1
kayasaman
Member
 
Registered: Sep 2008
Location: Under the bridge where proper engineers walkover
Distribution: Various Linux, Solaris, BSD, Cisco
Posts: 443

Rep: Reputation: 32
Strange HD space crunch - could it be ZFS???


Hi,

I'm running Squid as a reverse proxy on a Sun Netra T105 SPARC machine with an 18GB HD.

It's been running perfectly fine since I initially built it back in Feb/March time. Now all of a sudden I have no more space left on the hard drive???

Initially I thought it could be that Squid's cache got too large however after running du -sh cache under /opt/csw/var/ the system reported hardly any space taken up at all. Since then I have rebuilt the cache numerous times and each resulted in a few hundred megs freed up but nothing substantial... I also have deleted all the installer packages I got from SunFreeware in my home directory which also cleared a few hundred megs.

Squid now keeps going down because of no space left on the hard drive???

I have even changed the cache values from 90-95% to 20-25% as limits for the cache and it has had some effect but nothing substatial.

Can anyone help me figure out what's going on???

Here's some data that could be useful to start the figuring out process:

Code:
-bash-3.00# zfs list -o used 
 USED
16.4G
15.4G
15.4G
 512M
47.9M
47.9M
 518M

-bash-3.00# uname -a
SunOS NetraT1-Proxy 5.10 Generic_137137-09 sun4u sparc SUNW,UltraSPARC-IIi-cEngine

-bash-3.00# du -sh cache
  10M   cache
as can be seen hardly anything used up by Squid so where have the 170MB that I made yesterday gone???

Many thanks for any responses!!
 
Old 10-15-2009, 08:31 AM   #2
zhjim
Senior Member
 
Registered: Oct 2004
Distribution: Debian Squeeze x86_64
Posts: 1,748
Blog Entries: 11

Rep: Reputation: 233Reputation: 233Reputation: 233
I'm not in to Solaris in any way but if the squid cache is just that small I would look for nother reason. Maybe some log files stacking up? There sure is a logrotate utility!?
When I'm going out on space I normaly check on the disk space from / partition and then follow the path which holds the biggest space. There are also tools that graph the usage of certain dirs. Or mabye do a find with a size option...

Hope that helps in any way
 
Old 10-15-2009, 09:34 AM   #3
jlliagre
Moderator
 
Registered: Feb 2004
Location: Outside Paris
Distribution: Solaris 11.4, Oracle Linux, Mint, Tribblix, Ubuntu/WSL
Posts: 9,767

Rep: Reputation: 468Reputation: 468Reputation: 468Reputation: 468Reputation: 468
If you didn't reboot or at least restart the applications, it might be a large file still open while deleted at the directory level but it's hard to tell without more clues. Another explanation would be a snapshot retaining the deleted files bits.

You might want to post the output of these commands to figure out what's going on:
Code:
zpool status
zfs list
for i in $(zfs list -o mountpoint | egrep -v "(MOUNTPOINT|-)")
do
  du -hdsk $i
done
 
Old 10-15-2009, 10:02 AM   #4
Blinker_Fluid
Member
 
Registered: Jul 2003
Location: Clinging to my guns and religion.
Posts: 683

Rep: Reputation: 63
Had this forwarded to me the other day for large file space that won't release... (du and df reporting different values)

(as root)
cd /proc
ls -l */fd/*

look for a huge file listed
for example I have one that shows up like this:
-rw-r--r-- 0 webservd webservd 412925158 Oct 15 07:57 915/fd/18

The PID of the process holding the filespace is 915 at this point you can stop that process and then clear out space
(ps -ef | grep 915 and do whatever you need to for restarting and clearing space)
or cd /proc/915/fd and type :>18 to zero out the file if a restart of the process is not possible. (a restart of the processes is highly advised)

Last edited by Blinker_Fluid; 10-15-2009 at 10:31 AM.
 
Old 10-15-2009, 05:17 PM   #5
kayasaman
Member
 
Registered: Sep 2008
Location: Under the bridge where proper engineers walkover
Distribution: Various Linux, Solaris, BSD, Cisco
Posts: 443

Original Poster
Rep: Reputation: 32
Many thanks for all the responses;

jlliagre the outputs are as follows:

Code:
-bash-3.00# zpool status
  pool: rpool
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          c0t0d0s0  ONLINE       0     0     0

errors: No known data errors
Code:
-bash-3.00# zfs list
NAME                        USED  AVAIL  REFER  MOUNTPOINT
rpool                      16.5G  4.34M    94K  /rpool
rpool/ROOT                 15.4G  4.95M    18K  legacy
rpool/ROOT/s10s_u6wos_07b  15.4G  4.95M  15.4G  /
rpool/dump                  512M  4.78M   512M  -
rpool/export               47.9M  4.95M    20K  /export
rpool/export/home          47.9M  4.95M  47.9M  /export/home
rpool/swap                  518M  4.95M   518M  -
I tried running this also:

Code:
for i in $(zfs list -o mountpoint | egrep -v "(MOUNTPOINT|-)")
do
  du -hdsk $i
done
the machine subsequently produced this result then hung:

Code:
-bash-3.00# for i in $(zfs list -o mountpoint | egrep -v "(MOUNTPOINT|-)")
> do
>   du -hdsk $i
> done
  78K   /rpool
   0K   legacy
I waited for a bit and it did nothing so needed to ^Z it to break out!

The output from proc is as follows (see attach):
Attached Files
File Type: txt proc.txt (51.8 KB, 4 views)
 
Old 10-15-2009, 05:20 PM   #6
kayasaman
Member
 
Registered: Sep 2008
Location: Under the bridge where proper engineers walkover
Distribution: Various Linux, Solaris, BSD, Cisco
Posts: 443

Original Poster
Rep: Reputation: 32
I actually wasn't aware that I had mysql in here so it could be that somehow the database grew so large that it took up the whole HD although I don't even think it's running - also as another thing I don't even know where the database is stored lol.

Time to investigate a little I think!
 
Old 10-15-2009, 05:47 PM   #7
kayasaman
Member
 
Registered: Sep 2008
Location: Under the bridge where proper engineers walkover
Distribution: Various Linux, Solaris, BSD, Cisco
Posts: 443

Original Poster
Rep: Reputation: 32
Ok further investigation presents this:

Code:
-bash-3.00# du -sh *
  24K   TT_DB
   0K   bin
  86K   boot
   1K   cdrom
 255K   dev
 196K   devices
  69M   etc
  48M   export
   0K   home
  45M   kernel
  34M   lib
   4K   mnt
   0K   net
 2.4G   opt
 276M   platform
  16G   proc
 213M   root
  78K   rpool
 1.6M   sbin
 6.5M   system
   1K   tftpboot
 344K   tmp
 3.5G   usr
 8.9G   var
   0K   vol
Now proc seems to have colonized the system meaning that I will most likely have to reboot since as far as I know I'm not meant to touch /proc since that's generated specifically by the system itself.

I don't know if there's a way within the OS to actually check if openboot has:

auto-boot? true enabled??

Usually I would do this in front of the machine itself but since I am in another country currently and don't have a serial line server through which I can just SSH then run all my stty sessions I am a bit worried that issuing the reboot command will throw the system in openboot...

However even with the system restart will that clear the /proc dir or will it be the same... Help!!??
 
Old 10-15-2009, 05:52 PM   #8
kayasaman
Member
 
Registered: Sep 2008
Location: Under the bridge where proper engineers walkover
Distribution: Various Linux, Solaris, BSD, Cisco
Posts: 443

Original Poster
Rep: Reputation: 32
Quick conclusion it seems the offending app is Tomcat - although have run pkill tomcat I am not sure in Solaris how to see if it is still running??

In linux would run something like ps -aux | grep tomcat but here I can't....
 
Old 10-15-2009, 05:58 PM   #9
kayasaman
Member
 
Registered: Sep 2008
Location: Under the bridge where proper engineers walkover
Distribution: Various Linux, Solaris, BSD, Cisco
Posts: 443

Original Poster
Rep: Reputation: 32
ok running:

Code:
-bash-3.00# ps -ef | grep tomcat
  tomcat 24898     1   0 00:54:19 ?           0:18 /usr/java/bin/java -Djava.util.logging.config.file=/opt/csw/share/tomcat5/conf/
  tomcat 24883     1   4 00:54:11 ?           0:13 /usr/java/bin/java -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogMa
then ran kill 24898 only for another process to start again :-(
 
Old 10-15-2009, 07:49 PM   #10
kayasaman
Member
 
Registered: Sep 2008
Location: Under the bridge where proper engineers walkover
Distribution: Various Linux, Solaris, BSD, Cisco
Posts: 443

Original Poster
Rep: Reputation: 32
Few finally figured out what was causing the disk usage and it turned out to be the Sun Secure Global Desktop software!

Basically that runs a whole suite of programs such as java web server and tomcat6 so now have removed that package and now just in process of rebooting...

System has rebooted and now I have close to a gig left it seems but again /proc is filled???

It's still Tomcat? I have stopped it from even starting by edited the rc.conf files so I can't understand why at reboot it still is coming up...
 
Old 10-16-2009, 03:07 AM   #11
jlliagre
Moderator
 
Registered: Feb 2004
Location: Outside Paris
Distribution: Solaris 11.4, Oracle Linux, Mint, Tribblix, Ubuntu/WSL
Posts: 9,767

Rep: Reputation: 468Reputation: 468Reputation: 468Reputation: 468Reputation: 468
Quote:
Originally Posted by kayasaman View Post
I tried running this also:

Code:
for i in $(zfs list -o mountpoint | egrep -v "(MOUNTPOINT|-)")
do
  du -hdsk $i
done
the machine subsequently produced this result then hung:

Code:
-bash-3.00# for i in $(zfs list -o mountpoint | egrep -v "(MOUNTPOINT|-)")
> do
>   du -hdsk $i
> done
  78K   /rpool
   0K   legacy
I waited for a bit and it did nothing so needed to ^Z it to break out!
It *was* doing something, you should have been more patient ...
Quote:
Few finally figured out what was causing the disk usage and it turned out to be the Sun Secure Global Desktop software!

Basically that runs a whole suite of programs such as java web server and tomcat6 so now have removed that package and now just in process of rebooting...

System has rebooted and now I have close to a gig left it seems but again /proc is filled???
/proc isn't disk based but a virtual file system. There is no point trying to free it.
Quote:
It's still Tomcat? I have stopped it from even starting by edited the rc.conf files so I can't understand why at reboot it still is coming up...
ssgd can easily be turned off by running /etc/init.d/sun.com-sgd-base stop (might be a different name depending on your version).
You should also remove the start script that should be in /etc/rc3.d.
Finally, if you don't need ssgd, you can just uninstall its package ("pkginfo | grep tta" will tell you the package name to remove).
 
Old 10-16-2009, 04:52 AM   #12
kayasaman
Member
 
Registered: Sep 2008
Location: Under the bridge where proper engineers walkover
Distribution: Various Linux, Solaris, BSD, Cisco
Posts: 443

Original Poster
Rep: Reputation: 32
Many thanks for that!

However the space is still full??

Code:
-bash-3.00$ ps -ef
     UID   PID  PPID   C    STIME TTY         TIME CMD
    root     0     0   0 11:34:52 ?           0:01 sched
    root     1     0   0 11:34:53 ?           0:00 /sbin/init
    root     2     0   0 11:34:53 ?           0:00 pageout
    root     3     0   0 11:34:53 ?           0:01 fsflush
    root   478   474   0 11:36:37 ?           0:00 /usr/sadm/lib/smc/bin/smcboot
    root     7     1   0 11:34:56 ?           0:10 /lib/svc/bin/svc.startd
    root     9     1   0 11:34:57 ?           0:52 /lib/svc/bin/svc.configd
    root   230     1   0 11:36:02 ?           0:00 /bin/sh /opt/csw/mysql5/bin/mysqld_safe --pid-file=/opt/csw/mysql5/var/mysql.pi
   smmsp   635     1   0 11:37:23 ?           0:00 /usr/lib/sendmail -Ac -q15m
    root    74     1   0 11:35:22 ?           0:00 devfsadmd
    root   469     7   0 11:36:37 console     0:00 /usr/lib/saf/ttymon -g -d /dev/console -l console -m ldterm,ttcompat -h -p Netr
    root   985   984   0 11:38:19 ?           0:00 /opt/SUNWut/lib/utmountd
    root   895     1   0 11:38:01 ?           0:00 /usr/lib/inet/in.dhcpd
    root   636     1   0 11:37:23 ?           0:00 /usr/lib/snmp/snmpdx -y -c /etc/snmp/conf
  daemon   376     1   0 11:36:28 ?           0:00 /usr/lib/nfs/nfsmapid
  daemon   373     1   0 11:36:25 ?           0:00 /usr/sbin/rpcbind
    root   151     1   0 11:35:40 ?           0:00 /usr/sbin/nscd
    root   603     1   0 11:37:17 ?           0:00 /usr/sbin/syslogd
    root   133     1   0 11:35:39 ?           0:00 /usr/lib/power/powerd
    root   147     1   0 11:35:40 ?           0:00 /usr/lib/picl/picld
    root   119     1   0 11:35:37 ?           0:00 /usr/lib/sysevent/syseventd
  daemon   126     1   0 11:35:37 ?           0:01 /usr/lib/crypto/kcfd
    root   583     1   0 11:37:13 ?           0:00 /usr/lib/autofs/automountd
    root   403   400   0 11:36:32 ?           0:00 /usr/lib/saf/ttymon
    root   411     1   0 11:36:33 ?           0:03 /usr/lib/inet/inetd start
    root   207     1   0 11:35:57 ?           0:00 /usr/lib/inet/xntpd
  daemon   396     1   0 11:36:31 ?           0:00 /usr/lib/nfs/nfs4cbd
    root   231     1   0 11:36:02 ?           0:00 /bin/ksh -p /etc/opt/SUNWutsvt/utsvtd start
    root   400     7   0 11:36:32 ?           0:00 /usr/lib/saf/sac -t 300
    root   474     1   0 11:36:37 ?           0:00 /usr/sadm/lib/smc/bin/smcboot
  daemon   383     1   0 11:36:29 ?           0:00 /usr/lib/nfs/statd
    root   608     1   0 11:37:18 ?           0:09 /usr/lib/fm/fmd/fmd
    root   987   986   0 11:38:19 ?           0:00 /opt/SUNWut/lib/utstoraged -r
   mysql   369   230   0 11:36:24 ?           0:02 /opt/csw/mysql5/libexec/mysqld --basedir=/opt/csw/mysql5 --datadir=/opt/csw/mys
    root   299   231   0 11:36:08 ?           0:00 sleep 86400
    root  1373   598   0 11:41:47 ?           0:00 /usr/lib/ssh/sshd
    root   624     1   0 11:37:20 ?           0:01 /usr/sfw/sbin/snmpd
    root   242     1   0 11:36:03 ?           0:00 /usr/sbin/cron
   proxy   908   905   0 11:38:02 ?           0:00 (squid) -D
    root   598     1   0 11:37:16 ?           0:00 /usr/lib/ssh/sshd
    root   905     1   0 11:38:02 ?           0:00 /opt/csw/sbin/squid -D
    root   584   583   0 11:37:13 ?           0:00 /usr/lib/autofs/automountd
    root   645     1   0 11:37:25 ?           0:00 /usr/apache/bin/httpd -f /etc/apache/httpd-standalone-ipp.conf
    root   475   474   0 11:36:37 ?           0:00 /usr/sadm/lib/smc/bin/smcboot
  daemon   451     1   0 11:36:35 ?           0:00 /usr/lib/nfs/lockd
    root   452     1   0 11:36:35 ?           0:00 /usr/lib/utmpd
    root   637     1   0 11:37:23 ?           0:00 /usr/lib/sendmail -bd -q15m
    root   899     1   0 11:38:01 ?           0:00 /usr/lib/inet/dsvclockd
    root   633     1   0 11:37:22 ?           0:00 /usr/lib/lp/local/lpsched
    root   667     1   0 11:37:28 ?           0:00 /usr/lib/dmi/snmpXdmid -s NetraT1-Proxy
    root   984     1   0 11:38:19 ?           0:00 /bin/ksh /opt/SUNWut/lib/utlog -o /var/opt/SUNWut/log/utmountd.log
   proxy   919   908   0 11:38:03 ?           0:00 (unlinkd)
      lp   646   645   0 11:37:25 ?           0:00 /usr/apache/bin/httpd -f /etc/apache/httpd-standalone-ipp.conf
    root   659     1   0 11:37:27 ?           0:00 /usr/lib/dmi/dmispd
    root   680     1   0 11:37:30 ?           0:00 /usr/sbin/vold -f /etc/vold.conf
    kaya  1378  1376   0 11:41:52 syscon      0:00 -bash
    root   913     1   0 11:38:03 ?           0:00 /usr/dt/bin/dtlogin -daemon
    root   986     1   0 11:38:19 ?           0:00 /bin/ksh /opt/SUNWut/lib/utlog -o /var/opt/SUNWut/log/utstoraged.log
  tomcat  1436     1  20 11:42:30 ?           0:16 /usr/java/bin/java -Djava.util.logging.config.file=/opt/csw/share/tomcat5/conf/
  tomcat  1409     1  47 11:42:19 ?           0:29 /usr/java/bin/java -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogMa
    kaya  1439  1378   1 11:43:10 syscon      0:00 ps -ef
    kaya  1376  1373   1 11:41:52 ?           0:00 /usr/lib/ssh/sshd
    root   994   987   0 11:38:19 ?           0:00 /opt/SUNWut/lib/utstoraged -r
    root  1077     1   0 11:38:25 ?           0:00 /opt/SUNWuttsc/lib/uttscpd
rc3.d, I don't see the startup script as ran pkgrm tta on it:

Code:
-bash-3.00# ls
K50apache       README          S80mipagent     S98utstorage    s99utwadmin
K50cswapache    S16boot.server  S81uttsquantum  S99uttscp
K51tomcat6      S52imq          S84appserv      s98utsvc

cd /proc ls -lh */fd/* gives me this:

Code:
--w-------   1 tomcat   tomcat      6.6G Oct 16 11:47 1688/fd/1
--w-------   1 tomcat   tomcat      6.6G Oct 16 11:47 1688/fd/2
which I think is tied into here:

Code:
-bash-3.00# ps -ef | grep tomcat
  tomcat  1825     1  21 11:50:05 ?           0:09 /usr/java/bin/java -Djava.util.logging.config.file=/opt/csw/share/tomcat5/conf/
  tomcat  1798     1  29 11:49:54 ?           0:15 /usr/java/bin/java -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogMa
If I can get rid of this process I'll be ok!!
 
Old 10-16-2009, 05:52 AM   #13
kayasaman
Member
 
Registered: Sep 2008
Location: Under the bridge where proper engineers walkover
Distribution: Various Linux, Solaris, BSD, Cisco
Posts: 443

Original Poster
Rep: Reputation: 32
Finally.... I managed to sort this thing out :-D

Basically removed tomcat from the system:

pkgrm CSWtomcat5
pkgrm CSWtocmat6

then rebooted; this at least stopped /proc from being filled - from there ran du -sh * on / {root}

of which after traced the filled space back to /var/opt/csw, there were 2 directories in there called tomcat5 and tomcat6 respectively which I rm -rf'ed and now is all ok got 9.5GB available :-)

Many thanks everyone for all the support and help!!!! I really appreciate that :-)
 
Old 10-16-2009, 05:56 AM   #14
jlliagre
Moderator
 
Registered: Feb 2004
Location: Outside Paris
Distribution: Solaris 11.4, Oracle Linux, Mint, Tribblix, Ubuntu/WSL
Posts: 9,767

Rep: Reputation: 468Reputation: 468Reputation: 468Reputation: 468Reputation: 468
Note that this instance of tomcat was unrelated to ssgd or any software Sun ships.
 
Old 10-16-2009, 07:07 AM   #15
kayasaman
Member
 
Registered: Sep 2008
Location: Under the bridge where proper engineers walkover
Distribution: Various Linux, Solaris, BSD, Cisco
Posts: 443

Original Poster
Rep: Reputation: 32
Quote:
Note that this instance of tomcat was unrelated to ssgd or any software Sun ships.
Yes.... it was 3rd party as CSW indicates Blastwave!

I think I tried to use the server as both reverse proxy and remote desktop server with a whole bunch of extra stuff, however at the time I was even less experienced with Solaris/Sun and so I made a lot of mistakes and the biggest one was with assuming that the Sun Netra T105 server with a 360MHz CPU and 512MB RAM could take so much that I loaded on it!!

Also since the server is relatively low powered I should have just proceeded with a minimal install and used it as nothing but a reverse proxy.

All sorted now and have managed to clear out any unnecessary processes and software services so the system should at least perform a little bit better :-)
...less memory usage and definitely not as much HD space being taken up.

jlliagre I mean at least I think I am a lot better then when I started as you where one of the first people to help me out :-) - still lots of work to do though!
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Chinese Linux hit by credit crunch LXer Syndicated Linux News 0 12-11-2008 03:40 PM
Why the Disk Workout (Crunchy Crunch)? GTrax Linux - General 2 03-18-2007 05:18 AM
ZFS Root / Boot into ZFS from a usb flash drive Kataku Solaris / OpenSolaris 1 07-15-2006 05:13 AM
Strange Disk Space Issues cj10111 Red Hat 6 06-10-2005 05:24 AM
LFS (another q from the crunch) Crunch Linux - Newbie 6 02-22-2003 02:58 PM

LinuxQuestions.org > Forums > Other *NIX Forums > Solaris / OpenSolaris

All times are GMT -5. The time now is 11:48 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration