LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General > LinuxQuestions.org Member Success Stories
User Name
Password
LinuxQuestions.org Member Success Stories Just spent four hours configuring your favorite program? Just figured out a Linux problem that has been stumping you for months?
Post your Linux Success Stories here.

Notices

Reply
 
Search this Thread
Old 07-14-2009, 08:24 AM   #1
acid_kewpie
Moderator
 
Registered: Jun 2001
Location: UK
Distribution: Gentoo, RHEL, Fedora, Centos
Posts: 43,397

Rep: Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963
How to kill a server with logrotate


Howdy y'all.

I had to do some sleuthing on an inherited server and thought someone out there might find it interesting...

We have recently been given a half dozen RHEL & Oracle RAC servers to run and a few weeks into production we're getting automatic tickets raised about excession usage of /dev/sda on a system that's only running RAC from a NAS, suggesting minimal disk activity should occur locally.

So looking at the reports we see a spike, or a table-top mountain, in disk IO between about 03:50 and 04:20 every morning. Sound like a familiar time? cron.daily! But there was nothing interesting in the cron directory, or /etc/crontab, but the personal root crontab actually contains a "logrotate -f oraclelogs" kicking off around 3:40, so there we go... Why is that there?? Why didn't whatever muppet implemented that use cron.daily?? Slack. Now looking at the logrotate file there are about 20 different directories, the entire contents of each is being rotated and eventually we find one single log directory containing 540000 files. Roughly. Nice.

These log files are called something like css84952.log where the number appears to be a pid. And these appear to be client connections which spawn a new process to do its thing, so each connection creates a new, countably unique, log file with 2 lines in it.

Now about here I start laughing as the default logrotate config uses the "create" directive, touching a new file in place of the one it just rotated and gzipped.

SO... joining the dots, every day approximately 200-300 new log files were created, with minimal content. And due to the "create" happening, once a file exists, it will always exist forevermore, and then have 8 files per original file to process, with the whole number of files, and work, getting bigger and bigger and bigger everyday, until the server would probably just run out of inodes on that filesystem.

Solutions... add "nocreate" to this config file, and kick whoever set this up in the first place in the nuts. And tell them to NEVER use wildcards in a logrotate file!

There may well have been an Oracle level solution too, but that's none of my business really...

Last edited by acid_kewpie; 07-14-2009 at 08:28 AM.
 
Old 07-15-2009, 05:30 PM   #2
OdinnBurkni
Member
 
Registered: Feb 2007
Location: Iceland
Distribution: Fedora 14, CentOS, FreeNAS
Posts: 126

Rep: Reputation: 20
Logrotate

Thanks man...
I haven't had this issue but it makes you think outside the box...
It's a great thing to have a post like this, just to tell us what we might face one day or maybe just to help us not to make that mistake.
 
Old 07-15-2009, 05:43 PM   #3
anomie
Senior Member
 
Registered: Nov 2004
Location: Texas
Distribution: RHEL, Scientific Linux, Debian, Fedora, Lubuntu, FreeBSD
Posts: 3,930
Blog Entries: 5

Rep: Reputation: Disabled
We're a RHEL + Oracle RAC shop too. Good to be aware of.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Kill X Server Sintu Linux - Newbie 11 12-27-2006 12:11 AM
trying to kill a process on Red Hat server basskleff Linux - General 1 10-02-2006 05:01 PM
Cron.daily sends me a logrotate Kill usage message. pollardd Linux - General 1 09-02-2005 12:18 PM
what can i kill , web server using too much resources maxboost Linux - Enterprise 3 02-23-2005 04:53 PM


All times are GMT -5. The time now is 12:53 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration