monitoring disk space

cuss · 02-18-2003, 10:20 AM

Hi,

Does anyone know if logs are automatically generated when disk space reaches a certain threshold (ie. 80%)? I would like to be alerted when disk space reaches 80% with a log watching program named SWATCH. I already have SWATCH monitoring many things for me. If logs are generated at a certain threshold can anyone tell me the output of the logfile? SWATCH is configured by looking at certain patterns in a log file. I have a config file setup to 'watchfor /file system full/', but i'd like to have it setup something like 'watchfor /80%/ (this all depends on the output of the logfile so i know what to extract from it for SWATCH to monitor).

Thanks.

PTBmilo · 02-18-2003, 03:30 PM

You could make a cron job of the 'df' command... something like:

#!/bin/bash
df -h /home > /var/log/disk_space.log

and have you SWATCH program check that.

cuss · 02-18-2003, 03:56 PM

What type of log would be generated from that? Do you perhaps have a sample log from 'disk_space.log' that you could share? If I wanted cron to check disk space every 60 minutes I assume I would have to add the appropriate entries to the cron job? Also, wouldn't I actually place the above script somewhere else and then write a cron job to check that particular script every 60 minutes. Or will one script in crontab as illustrated above be sufficient? And then i just write a config file for SWATCH to monitor for 80% disk usage based on the output of the log file.

Thanks!

PTBmilo · 02-18-2003, 08:16 PM

Well, I haven't really played around with cron that much, so I can't help with the specific syntax for the configuration, but you have the idea.

Put what's up there in a text file, and make it executable with:

chmod 755 disk_free_logger.sh (or whatever you wanna call it)

Then run that script from cron. On my slack box, it looks like you just put it in /etc/cron.hourly/ , but I'm not sure if that's the standard (I don't think so)

Just play around with the command so that it gives you what you want. Here's what I'd get:

Code:

/home/milo [219]> df -h /home > disk_free.log
/home/milo [220]> cat disk_free.log 
Filesystem            Size  Used Avail Use% Mounted on
/dev/hdb1             1.8G  1.4G  378M  79% /

As you can see, I'm almost to the 80% limit

A somewhat more elegant log could be created with something like:

Code:

#!/bin/bash
date >> /var/log/disk_free.log
df -h /home | cut -c1-9,40-43 | grep /dev >> /var/log/disk_free.log

using the '>>' it will append everything to the end of the file, while if there's a single '>', it will erase everything else.... you can choose what you want to do. Here's what I get from that:

Code:

/home/milo [267]> date > disk_free.log                                         
/home/milo [268]> df -h /home | cut -c1-9,40-43  | grep /dev >> disk_free.log
/home/milo [269]> cat disk_free.log 
Tue Feb 18 12:14:56 PST 2003
/dev/hdb1 79%

Just play around with it till you get what you want in the log file.

Have fun

-Milo

PTBmilo · 02-18-2003, 08:38 PM

Ok, I'm really dorking out here now

check out this version (probably MUCH better ways to do it):

Code:

#!/bin/bash
date > /tmp/date_output.txt
df -h /home | cut -c1-9,40-43 | grep /dev | \
paste - /tmp/date_output.txt >> /var/log/disk_free.log

Here's the output:

Code:

root@temple:~# ./date_logger.sh 
root@temple:~# ./date_logger.sh 
root@temple:~# ./date_logger.sh 
root@temple:~# cat /var/log/disk_free.log 
/dev/hdb1 79%   Tue Feb 18 12:36:05 PST 2003
/dev/hdb1 79%   Tue Feb 18 12:37:06 PST 2003
/dev/hdb1 79%   Tue Feb 18 12:37:09 PST 2003
/dev/hdb1 79%   Tue Feb 18 12:37:11 PST 2003

the /tmp/date_output.txt will remain a one line text file... won't suck up disk space.

rnturn · 02-18-2003, 08:58 PM

Quote:

Originally posted by PTBmilo
Ok, I'm really dorking out here now

Hah! I think I outdid you! (Out dorked?) Several years ago, I sat down and wrote a pair of perl scripts: one that collect usage data on all the filesystems and saves the most recent measurements and a second script that: 1) if run interactively shows a simple summary of filesystem, space, and whether free space is growing or shrinking or 2.) if run non-interactively does a least-squares fit to the samples and makes a prediction as to when the filesystem(s) will run out of space and, if it's in less than a certain number of days, it sends me an email. Other users can receive emails about certain filesystems that they care about, the thresholds can be set by filesystem, etc..

It all started out to be a much simpler monitor but I got on a roll... :-)

PTBmilo · 02-18-2003, 11:59 PM

OK, I got sucked into my shell programing book (I needed to study anywase).

This version will check all of your partitions, and only log the ones that are over (and equal to) 80% used.

Code:

#!/bin/bash
for i in $( fdisk -l | cut -c1-9 | grep /dev/ ) # Get all partitions
do
        if [ "$( df -h $i | grep -v U | cut -c40-42 )" -ge "80" ]; then

                date > /tmp/date_output.txt

                df -h $i | cut -c1-10,40-43 | grep /dev | \
                paste - /tmp/date_output.txt >> /var/log/disk_free.log
        fi
done

and here's my output:

Code:

/dev/hda5  93%  Tue Feb 18 15:50:34 PST 2003
/dev/hdb2 100%  Tue Feb 18 15:50:34 PST 2003

That freak'n integer test operation ( -ge NOT > ) had me scratching my head for a while.

I'LL GET YOU NEXT TIME RNTURN... I'LL GET YOU!!!

RRRREEEEEOOOOWWWW!!!!!

<attempt to emulate 'inspector gadget' closing lines>

really though man... you need help

cuss · 02-19-2003, 11:47 AM

Thanks PTBmilo!!

I finally had some time today to try your last script and with a couple very minor modifications it works beautifully! I now have cron checking my disk_space.sh file (your script) every 60 minutes and have SWATCH monitoring the logfile output for anything greater than 80%.

Thanks again.

acid_kewpie · 02-19-2003, 11:49 AM

Code:

#!/bin/bash
for i in $( fdisk -l | cut -c1-9 | grep /dev/ ) # Get all partitions
do
        if [ "$( df -h $i | grep -v U | cut -c40-42 )" -ge "80" ]; then

                df -h $i | cut -c1-10,40-43 | grep /dev | \
                paste - `date` >> /var/log/disk_free.log
        fi
done

it's a fairly nice approach (i expect there's an official daemon out there somewhere but i've no idea....) i'd suggest not writing it to a file at all but either using a wall command to send the message to every open terminal when that happens, or have it update /etc/motd (or just have the code directly in your .bashrc) instead, to give a quick list of stats each time a new terminal window is opened.

PTBmilo · 02-19-2003, 04:19 PM

Quote:

Originally posted by acid_kewpie

Code:

paste - `date` >> /var/log/disk_free.log

Thanks Acid... I had a feeling I was missing something by needing that temp. file.

Code:

echo "`df -h $i | cut -c1-10,40-43 | grep /dev` `date`" \
>> /var/log/disk_free.log

BryanMC · 02-19-2003, 11:21 PM

what shell programming book are you reading PTBmilo??

I'm a newbee but I have some programming experience.....

crichards · 02-19-2003, 11:49 PM

If you want to make a more automated, try using this on the last line.

Code:

| awk '{print $2}'

That will print just the percentages. You could write a small script to take logs for each drive and check if its out of space, and you wouldn't have to worry about all of the extra stuff like the drive name and such. I'd use a for loop to do it all.

PTBmilo · 02-20-2003, 01:42 AM

BryanMC, I have:
"UNIX shell programming" (revised edition)
Stephen G. Kochan, Patrick H. Wood

I like it a lot so far... It's not too wordy, just gets to the point in an easy to follow manner.

Quote:

Originally posted by crichards
You could write a small script to take logs for each drive and check if its out of space, and you wouldn't have to worry about all of the extra stuff like the drive name and such. I'd use a for loop to do it all.

crichards · 02-20-2003, 09:41 AM

Sorry, I didn't make that clear enough. You could set up the aforementioned cron jobs, and make it output one file for each drive. Then, you could have the script check to see if the last line was equal to or greater than 80%. Thats what I meant, didn't make it clear...

Sorry.

I don't exactly know what I was thinking.

cuss · 02-20-2003, 09:55 AM

Hi,

I implemented the script and started it with cron on another server but with no luck. To test I set the '-ge "80"' value in the original script to '-ge "0"' so that all partitions would send a log to /var/log/disk_free.log (0 representing any partition over and equal to 0% disk space). The message i got from cron is as follows:

----------------------------------------------------------------------------------
Usage: fdisk [-l] [-b SSZ] [-u] device
E.g.: fdisk /dev/hda (for the first IDE disk)
or: fdisk /dev/sdc (for the third SCSI disk)
or: fdisk /dev/eda (for the first PS/2 ESDI drive)
or: fdisk /dev/rd/c0d0 or: fdisk /dev/ida/c0d0 (for RAID devices)
...
----------------------------------------------------------------------------------

I have 6 partitions, 5 of which are '/dev/hdc' and one which is '/dev/sda'. From the script it looks like anything in '/dev' should be logged. No? I checked /var/log/disk_free.log and there were no logs created. Can anyone help? The script works very nicely on my other server where all partions are '/dev/hda'.

Thanks.