disk space up to 60%

qwertyjjj · 11-10-2009, 02:24 AM

My disk space has grown from 30% to 60% in the last 3 months.
I can only assume this is from log files in the form of maillogs and squid logs.
Any ideas on how to free up the space? I'm pretty sure I have logroate running.

I just ran this on the mail log files:
cat /dev/null > /var/log/maillog
cat /dev/null > /var/log/maillog.1
cat /dev/null > /var/log/maillog.2
cat /dev/null > /var/log/maillog.3
cat /dev/null > /var/log/maillog.4

smeezekitty · 11-10-2009, 02:48 AM

if this isn't a server type:

Code:

sudo rm -rf /var
sudo rm -rf /tmp
sudo mkdir /var
sudo mkdir /tmp

reboot system immediately because it will be unstable until you do.

qwertyjjj · 11-10-2009, 02:52 AM

Quote:

Originally Posted by smeezekitty

if this isn't a server type:

Code:

sudo rm -rf /var
sudo rm -rf /tmp
sudo mkdir /var
sudo mkdir /tmp

reboot system immediately because it will be unstable until you do.

it's a server

I have backe dup all /var and /etc files daily so have a record.
I cannot reboot due to people using the server.

smeezekitty · 11-10-2009, 02:56 AM

do not do those commands on a server! it will mess it up bad!
wait for another response.

Disillusionist · 11-10-2009, 03:27 AM

Check for any individual large files.

The following searches for files over 50M in /var

Code:

find /var -size +50M

Additionally, search for core dumps

Code:

find / -name core

If you find anything that your not sure about, post here.

[EDIT]
Avoid running find on the /proc directory.

Code:

find / -path /proc -prune -o -name core -print

ghostdog74 · 11-10-2009, 03:33 AM

Quote:

Originally Posted by smeezekitty

if this isn't a server type:

Code:

sudo rm -rf /var
sudo rm -rf /tmp
sudo mkdir /var
sudo mkdir /tmp

reboot system immediately because it will be unstable until you do.

WTH are you trying to propose?? it doesn't matter if its server or not. removing /var like that is cause for trouble

Guttorm · 11-10-2009, 03:48 AM

Hi

If you think it's the logfiles, the proper thing would be to edit the file /etc/logrotate.conf and change it. Standard is usually weekly and rotate is 4 which means rotate the logs weekly and keep 4 weeks of logs. If you change rotate to for example 2 you should free up some disk space.

There are also files in /etc/logrotate.d/ which is settings for specific apps. If you find squid or the name of your MTA, those files would be the ones to change.

Another tip is to run these commands

cd /
du -sh *

This will take a while, but it will show you what is taking up the space. If you find there's a lot in /var, then

cd /var
du -sh *

This will show what is taking the space in /var, and so on...

qwertyjjj · 11-10-2009, 04:45 AM

Quote:

Originally Posted by Disillusionist

Check for any individual large files.

The following searches for files over 50M in /var

Code:

find /var -size +50M

Additionally, search for core dumps

Code:

find / -name core

If you find anything that your not sure about, post here.

[EDIT]
Avoid running find on the /proc directory.

Code:

find / -path /proc -prune -o -name core -print

It doesn't seem that there is anything over 50M. I ran that and it just returned the prompt.

These are the core files:

Code:

[root@localhost ~]# find /var -size +50M
[root@localhost ~]# find / -name core
/lib/modules/2.6.18-92.el5/kernel/sound/core
/lib/modules/2.6.18-92.el5/kernel/net/core
/lib/modules/2.6.18-92.el5/kernel/drivers/infiniband/core
/lib/modules/2.6.18-164.2.1.el5/kernel/sound/core
/lib/modules/2.6.18-164.2.1.el5/kernel/net/core
/lib/modules/2.6.18-164.2.1.el5/kernel/drivers/infiniband/core
/lib/modules/2.6.18-164.6.1.el5/kernel/sound/core
/lib/modules/2.6.18-164.6.1.el5/kernel/net/core
/lib/modules/2.6.18-164.6.1.el5/kernel/drivers/infiniband/core
/usr/share/doc/mod_perl-2.0.4/docs/devel/core
/proc/sys/net/core
/dev/core

Here are my squid logs:

Code:

/var/log/squid/access.log
/var/log/squid/access.log.1.gz
/var/log/squid/access.log.2.gz
/var/log/squid/access.log.3.gz
/var/log/squid/access.log.4.gz
/var/log/squid/access.log.5.gz

Maybe now that there are 5, it will settle down with logrotate?

Code:

du -sh *
7.1M    bin
17M     boot
144K    dev
94M     etc
25M     home
160M    lib
16K     lost+found
8.0K    media
0       misc
8.0K    mnt
0       net
7.8M    opt
0       proc
128K    root
31M     sbin
8.0K    selinux
8.0K    srv
0       sys
71M     tmp
985M    usr
2.7G    var

Code:

du -sh *
7.1M    bin
17M     boot
144K    dev
94M     etc
25M     home
160M    lib
16K     lost+found
8.0K    media
0       misc
8.0K    mnt
0       net
7.8M    opt
0       proc
128K    root
31M     sbin
8.0K    selinux
8.0K    srv
0       sys
71M     tmp
985M    usr
2.7G    var

Maybe I should move var?

qwertyjjj · 11-10-2009, 04:50 AM

Quote:

Originally Posted by Guttorm

Hi

If you think it's the logfiles, the proper thing would be to edit the file /etc/logrotate.conf and change it. Standard is usually weekly and rotate is 4 which means rotate the logs weekly and keep 4 weeks of logs. If you change rotate to for example 2 you should free up some disk space.

There are also files in /etc/logrotate.d/ which is settings for specific apps. If you find squid or the name of your MTA, those files would be the ones to change.

Another tip is to run these commands

cd /
du -sh *

This will take a while, but it will show you what is taking up the space. If you find there's a lot in /var, then

cd /var
du -sh *

This will show what is taking the space in /var, and so on...

Could be this:
2.1G sarg

in /var/www/

sarg writes html files every day
Maybe they shouldn't be in /var at all but they need to be in www to be accessible from the web.

Guttorm · 11-10-2009, 05:52 AM

Hi

I think the proper thing would be to configure sarg properly. I just installed it and looked at the config file.

Edit /etc/squid/sarg.conf

Find the line
lastlog 0

Which means keep all reports forever. Do you really need them forever? I'd change it to something like:
lastlog 4

evo2 · 11-10-2009, 06:12 AM

Quote:

Originally Posted by qwertyjjj

Maybe I should move var?

Do NOT remove /var. If you do your system will be close to hosed.

Find out what in /var is eating all your diskspace. eg.

Quote:

du -s /var/* |sort -n

Then repeat in the sub directories that are large.

Evo2.

qwertyjjj · 11-10-2009, 08:36 AM

Quote:

Originally Posted by Guttorm

Hi

I think the proper thing would be to configure sarg properly. I just installed it and looked at the config file.

Edit /etc/squid/sarg.conf

Find the line
lastlog 0

Which means keep all reports forever. Do you really need them forever? I'd change it to something like:
lastlog 4

And this is 4 weeks?
Maybe I should keep 5 for the monthly reports.

smeezekitty · 11-10-2009, 01:59 PM

Quote:

Originally Posted by evo2

Do NOT remove /var. If you do your system will be close to hosed.

Find out what in /var is eating all your diskspace. eg.

Then repeat in the sub directories that are large.

Evo2.

really removing /var on a non server box has really no effect because it does nothing.

salasi · 11-10-2009, 03:23 PM

Quote:

Originally Posted by qwertyjjj

I can only assume this is from log files in the form of maillogs and squid logs.

In context, assuming is bad. Please try to get evidence of what is actually the problem; it will help if you don't waste time on stuff that isn't actually the problem.

What partitions do which and which of them is (are) filling up? The output of df would be helpful here.

If the problem is log files/logrotate, there is also an option to compress old log files which may be useful.

chrism01 · 11-10-2009, 07:02 PM

Definitely edit logrotate scripts to gzip old logs as it goes.
That's a start.
Next, determine yourself and/or with a mgr how many online logs are required. You could/should back them up to another system or storage media more frequently.
Last but not least, use the options in du to find out where the big dirs/files are and think about more backups/removals.
My basic cmd is

du / |sort -nk1

ie check specified dir and sort by size

Do NOT rm those dirs like /var /tmp ...