pacct going wild with logs

richinsc · 06-15-2009, 08:18 AM

I have a SUSE system that is going wild with the logs. The logs were being rotated at one time but they no longer are. The disk is only 2.0GB, I can't take the server down because this is a live server that requires 24/7 operation and a downtime of no more then 5min per year. I can't drop to single user mode, because this system is accessed by users at all times, even during night. Below you will see that I have one file that refuses to rotate.

Code:

total 1.3G
drwxr-xr-x   2 root root  824 2009-06-09 08:16 .
drwxr-xr-x  23 root root  584 2009-06-15 09:02 ..
-rw-r-----   1 root root 994M 2009-06-15 03:45 pacct
-rw-r-----   1 root root 2.7M 2009-05-02 04:15 pacct-20090501.gz
-rw-r-----   1 root root 2.8M 2009-05-03 04:15 pacct-20090502.gz
-rw-r-----   1 root root 1.2M 2009-05-04 04:15 pacct-20090503.gz
-rw-r-----   1 root root 1.7M 2009-05-05 04:15 pacct-20090504.gz
-rw-r-----   1 root root 4.3M 2009-05-06 04:15 pacct-20090505.gz
-rw-r-----   1 root root 6.5M 2009-05-07 04:15 pacct-20090506.gz
-rw-r-----   1 root root 6.5M 2009-05-08 04:15 pacct-20090507.gz
-rw-r-----   1 root root 4.0M 2009-05-09 04:15 pacct-20090508.gz
-rw-r-----   1 root root 4.0M 2009-05-10 04:15 pacct-20090509.gz
-rw-r-----   1 root root 1.4M 2009-05-11 04:15 pacct-20090510.gz
-rw-r-----   1 root root 1.8M 2009-05-12 04:15 pacct-20090511.gz
-rw-r-----   1 root root 3.3M 2009-05-13 04:15 pacct-20090512.gz
-rw-r-----   1 root root 4.8M 2009-05-14 04:15 pacct-20090513.gz
-rw-r-----   1 root root 4.4M 2009-05-15 04:15 pacct-20090514.gz
-rw-r-----   1 root root 5.1M 2009-05-16 04:15 pacct-20090515.gz
-rw-r-----   1 root root 8.5M 2009-05-17 04:15 pacct-20090516.gz
-rw-r-----   1 root root 2.1M 2009-05-18 04:15 pacct-20090517.gz
-rw-r-----   1 root root 1.8M 2009-05-19 04:15 pacct-20090518.gz
-rw-r-----   1 root root 184M 2009-05-19 04:15 pacct-20090519

Here is what I put in logrotate. I recently modified and removed the dateext because it was driving me crazy anyway but I know it has nothing to do with the logs are they are now. I also changed to daily from weekly to try and force it to rotate. Executed logroate -f /etc/longrotate.conf and it still exists. So what is the proper way to get this problem taken care of and make disk space more manageable. I didn't build this system, so I don't know who in their right mind would have thought at 2gb for /var would have been sufficient for the user loads on this system. (It's a CVS repository)

Code:

/var/account/pacct {
    daily
    rotate 7
    size=+4096k
    compress
    delaycompress
    notifempty
    missingok
    create 640 root root
    postrotate
     /etc/init.d/acct force-reload
    endscript
}

However I think the reason why this is coming about is because sometime ago I had to change the logrotate and manually delete some of the older files, ie( there were files on here from 2008) I figured I would only need a month's worth of pacct logs. After manually removing those files I think that is when this came about but not sure.

unSpawn · 06-15-2009, 09:48 AM

Running 'logrotate -df /etc/logrotate.d/pacct 2>&1 | tee /tmp/logrotate.dbg' (or psacct?) should get you infos in /tmp/logrotate.dbg to look at (and post?) because I can't see anything wrong with this.

richinsc · 06-15-2009, 09:58 AM

This is the result after command

Code:

cat /tmp/logrotate.dbg
reading config file /etc/logrotate.d/acct
reading config info for /var/account/pacct
error: line too long in state file /var/lib/logrotate.status

richinsc · 06-15-2009, 10:03 AM

After seeing the line to long error I deleted the logrotate.status file, logrotate was able to execute and then I got this.

Code:

logrotate -df /etc/logrotate.d/acct 2>&1 | tee /tmp/logrotate.dbg
reading config file /etc/logrotate.d/acct
reading config info for /var/account/pacct
error: line too long in state file /var/lib/logrotate.status
solssvdev:/var/log# cat /tmp/logrotate.dbg
reading config file /etc/logrotate.d/acct
reading config info for /var/account/pacct
error: line too long in state file /var/lib/logrotate.status
solssvdev:/var/log# cat /tmp/logrotate.dbg
reading config file /etc/logrotate.d/acct
reading config info for /var/account/pacct
error: line too long in state file /var/lib/logrotate.status
solssvdev:/var/log# cp /var/lib/logrotate.status /root/
solssvdev:/var/log# rm /var/lib/logrotate.status
solssvdev:/var/log# logrotate -df /etc/logrotate.d/acct 2>&1 | tee /tmp/logrotate.dbg
reading config file /etc/logrotate.d/acct
reading config info for /var/account/pacct

Handling 1 logs

rotating pattern: /var/account/pacct  forced from command line (7 rotations)
empty log files are not rotated, old logs are removed
considering log /var/account/pacct
  log needs rotating
rotating log /var/account/pacct, log->rotateCount is 7
previous log /var/account/pacct.1 does not exist
renaming /var/account/pacct.7.gz to /var/account/pacct.8.gz (rotatecount 7, logstart 1, i 7),
renaming /var/account/pacct.6.gz to /var/account/pacct.7.gz (rotatecount 7, logstart 1, i 6),
renaming /var/account/pacct.5.gz to /var/account/pacct.6.gz (rotatecount 7, logstart 1, i 5),
renaming /var/account/pacct.4.gz to /var/account/pacct.5.gz (rotatecount 7, logstart 1, i 4),
renaming /var/account/pacct.3.gz to /var/account/pacct.4.gz (rotatecount 7, logstart 1, i 3),
renaming /var/account/pacct.2.gz to /var/account/pacct.3.gz (rotatecount 7, logstart 1, i 2),
renaming /var/account/pacct.1.gz to /var/account/pacct.2.gz (rotatecount 7, logstart 1, i 1),
renaming /var/account/pacct.0.gz to /var/account/pacct.1.gz (rotatecount 7, logstart 1, i 0),
renaming /var/account/pacct to /var/account/pacct.1
creating new log mode = 0640 uid = 0 gid = 0
running postrotate script
running script with arg /var/account/pacct: "
     /etc/init.d/acct force-reload
"
removing old log /var/account/pacct.8.gz

So that means I need to change it back to what it was and hope it works.

unSpawn · 06-15-2009, 10:05 AM

Good to see you fixed it yourself. I wonder though what caused the "line too long"?

richinsc · 06-15-2009, 10:09 AM

Ok after making the changes, it is rotating now and this is what I have in file.

Code:

/var/account/pacct {
    weekly
    dateext
    rotate 4
    size=+4096k
    compress
    delaycompress
    notifempty
    missingok
    create 640 root root
    postrotate
     /etc/init.d/acct force-reload
    endscript
}

However it still isn't rotating the 900mb file

Code:

ls -lh
total 1.1G
drwxr-xr-x   2 root root  832 2009-06-15 11:05 .
drwxr-xr-x  23 root root  584 2009-06-15 10:52 ..
-rw-r-----   1 root root 994M 2009-06-15 03:45 pacct
-rw-r-----   1 root root 2.7M 2009-05-02 04:15 pacct-20090501.gz
-rw-r-----   1 root root 2.8M 2009-05-03 04:15 pacct-20090502.gz
-rw-r-----   1 root root 1.2M 2009-05-04 04:15 pacct-20090503.gz
-rw-r-----   1 root root 1.7M 2009-05-05 04:15 pacct-20090504.gz
-rw-r-----   1 root root 4.3M 2009-05-06 04:15 pacct-20090505.gz
-rw-r-----   1 root root 6.5M 2009-05-07 04:15 pacct-20090506.gz
-rw-r-----   1 root root 6.5M 2009-05-08 04:15 pacct-20090507.gz
-rw-r-----   1 root root 4.0M 2009-05-09 04:15 pacct-20090508.gz
-rw-r-----   1 root root 4.0M 2009-05-10 04:15 pacct-20090509.gz
-rw-r-----   1 root root 1.4M 2009-05-11 04:15 pacct-20090510.gz
-rw-r-----   1 root root 1.8M 2009-05-12 04:15 pacct-20090511.gz
-rw-r-----   1 root root 3.3M 2009-05-13 04:15 pacct-20090512.gz
-rw-r-----   1 root root 4.8M 2009-05-14 04:15 pacct-20090513.gz
-rw-r-----   1 root root 4.4M 2009-05-15 04:15 pacct-20090514.gz
-rw-r-----   1 root root 5.1M 2009-05-16 04:15 pacct-20090515.gz
-rw-r-----   1 root root 8.5M 2009-05-17 04:15 pacct-20090516.gz
-rw-r-----   1 root root 2.1M 2009-05-18 04:15 pacct-20090517.gz
-rw-r-----   1 root root 1.8M 2009-05-19 04:15 pacct-20090518.gz
-rw-r-----   1 root root  11M 2009-06-15 11:05 pacct-20090519.gz

Result from log now

Code:

 cat /tmp/logrotate.dbg
reading config file /etc/logrotate.d/acct
reading config info for /var/account/pacct

Handling 1 logs

rotating pattern: /var/account/pacct  forced from command line (4 rotations)
empty log files are not rotated, old logs are removed
considering log /var/account/pacct
  log needs rotating
rotating log /var/account/pacct, log->rotateCount is 4
compressing log with: /bin/gzip
renaming /var/account/pacct to /var/account/pacct-20090615
creating new log mode = 0640 uid = 0 gid = 0
running postrotate script
running script with arg /var/account/pacct: "
     /etc/init.d/acct force-reload
"
removing old log /var/account/pacct-20090516.gz

richinsc · 06-15-2009, 10:16 AM

Oh Duh.. -d Don't do anything, just test (implies -v)

unSpawn · 06-15-2009, 11:06 AM

Quote:

Originally Posted by richinsc

Oh Duh.. -d Don't do anything, just test (implies -v)

Yes, it comes in handy when testing stuff without immediately b0rking things ;-p