Bash script help
 

Hello, all.

I'm creating a bash script to check how much free space is left in /var directory then, if it hits a certain threshold, delete certain files with numbers for extensions (e.g. fileA.1, fileA.2 fileA.3, and fileA.4, fileB.1, fileB.2 fileB.3, and fileB.4 ). Here's a snippet from my script:

if [ $PARTUSE -ge $ALERT ]; then
cat /dev/null > /var/log/fileA.* 2>&1 &&
cat /dev/null > /var/log/fileB.* 2>&1 &&
cat /dev/null > /var/log/fileC.* 2>&1 &&
echo "Reclaimed disk space on \"$PARTITION\" now ($PARTUSE% full) on $(hostname) as of $(date)" |
mail -s "Reclaimed disk space - $PARTITION on $(hostname)" $ADMIN
fi

If I use a * as a wildcard for the number extension, the script fails. Maybe regex would work here, but I'm not particularly accomplished at it. Or some other construct.

Your help would be most appreciated.

Diggy

You cannot redirect the standard output to multiple files at once. You can use a loop to empty every single file, e.g.
Note you don't need cat /dev/null. In bash/ksh the redirection symbol alone creates/overwrite a file as if the null command was executed. Hope this helps.

The colucix is the right approach...

Nope. To me it creates a literal fileA.* since escaping prevents expansion.

colucix,

I used a poor example (apologies) in decribing what I want to do. I really want to delete files along the lines of Afile.1, Afile.2, Bfile.1, Bfile.2. Given that, How would I write the construct you suggested if, in fact, it would still work in this case?

Diggy

And why you don't just use a rm command for that?
Too risky?
May it's not so clear (to me!) the phrase

sorry!

I don't want to destroy the files, just empty them of content.

Diggy

As colucix suggested should only empty the files, not delete them.

This actually empties the content of the file:
What are your doubts about the suggested loop?

colucix,

I'm sure that your solution 1) works, and 2) is "elegant". I (ashamedly) just don't have a lot of scripting experience. That said, would my entire script look like this?:

#!/bin/bash
#
# Shell script to reclaim disk space
# It will clear out certain files if the percentage of partition space used is >= % set in ALERT variable,
# thus reducing partition size, and will send an email to the system administrator(s)
# -------------------------------------------------------------------------
# Written by Me
# -------------------------------------------------------------------------
#
# set admin email address(es)
ADMIN="root@localhost"
#
# set alert level at 90%. If used space exceeds this, send alert
ALERT=80
#
# check partition space and, if necessary, send alert
df -H | grep '/var' | awk '{ print $5 " " $6 }' | while read output;
do
PARTUSE=$(echo $output | awk '{ print $1}' | cut -d'%' -f1 )
PARTITION=$(echo $output | awk '{ print $2 }' )
if [ $PARTUSE -ge $ALERT ]; then
for file in /var/log/Afile.? /var/log/Bfile.? /var/log/Bfile.?
do
> $file
done &&
echo "Reclaimed disk space on \"$PARTITION\" now ($PARTUSE% full) on $(hostname) as of $(date)" |
mail -s "Reclaimed disk space - $PARTITION on $(hostname)" $ADMIN
fi
done


Please bear with me, I'm learning, and this will be invaluable.

Diggy

Hey, no problem at all! I was just trying to clarify! :)

Regarding your script it looks good. I would avoid some redundancy, anyway. For example, to extract the usage of the /var partition you might use a more compact code. In place of:
you can simply do:
Notes:

1. the -H option of df is not really necessary, since you're not intersted in the actual sizes but only in the percentage. Instead the -P option is useful, since it causes the df output to be in a more standard format. There are situations where the device name is very long and a line of output can be splitted in two parts. The -P option prevents this behaviour.
2. awk is a very powerful language that manages regular expressions, so that the grep command is not necessary. Moreover it can remove the % sign, so that you can avoid the cut command.
3. the read statement accept multiple variable names. The input will be splitted to fill all the specified variables, so that you can assign PARTUSE and PARTITION simultaneously.

Finally I don't see the need for a while loop, since you're reading a single line. To assign the output of a command using the read statement, process substitution is your friend:
Hope this helps! :)

What you suggest looks just fine, but I'm not sure how the final product would look. Based on my knowledge, and the suggestions in your last post, I'm not seeing where/how the files are being concatenated.

I understand that this piece:

df -P | awk '/\/var/{ sub(/%/,"",$5); print $5, $6 }' | while read PARTUSE PARTITION
do

replaces:

df -H | grep '/var' | awk '{ print $5 " " $6 }' | while read output
do
PARTUSE=$(echo $output | awk '{ print $1}' | cut -d'%' -f1 )
PARTITION=$(echo $output | awk '{ print $2 }' )

After the "do" statement, do I place my email alert piece? What handles the actual file concatenation?

I'm a bit confused, and feeling not just a little stupid.

Diggy

It simply replaces the extraction of information from the df output and it does the assigment to PARTUSE and PARTITION in a different way, but the rest stays untouched:

I apologize, again. I didn't mean to have you write this for me. But, now I get it. I'll test in a while, and report back. Thanks for your patience!!

Diggy

You're welcome! :)

You don't need awk and all that to get the info out of the "df"; this shows how:

colucix,

It worked! It does raise another question for me, though.

I tried the same script on two machines. Machine 1 had files Afile.1 through .4, and Bfile.1 through .4, and Cfile.1 through .4. Machine 2 had only Afile.1 through .4. All of the files on Machine 1 were concatenated, just as I wanted. And, on Machine 2, Afile.1 through .4 were concatenated. But, files named Bfile.? and Cfile.? were created in the directory. Of course, I don't want that to happen.

So, in order to use the same script on different machines where any combination of Afiles, Bfiles, and Cfiles might exist, what should I do so that "missing" files are ignored. Hopefully, with modifications, your construct can be made to do that. BTW, the use of Afile, Bfile, and Cfile are purely illustrative. They could be named Curly.1, Moe.1, and Larry.1, etc.

As ever, thanks.

Diggy

To ignore "missing" files, you can do existence check like this:
In colucix's script, instead of
Try:

Ah, a conditional statement! Worked like a charm.

Thanks to you all for your help. This made my day!

Best,

Diggy

Another way is by means of the nullglob option:
From the bash reference manual:

If services log to file then logrotate is the answer (and not script) if you want to retain logging. If you do not want to retain logging then do not log (as much as you do now).

Good point, unSpawn! I was too focused on the shell issues to think about the logical thing to do. :doh:

UnSpawn of rkhunter fame? Excellent. I use that great program!

You're right about logrotate, but I need to use the script, regardless. Let me elucidate (finally got to use that word :-) ).

The box I'm running the script on is old - old OS, old hardware, small disk. Because so much log activity takes place, the /var directory fills up about every four weeks because of the growth of certain log files. Logrotate rotates them fine, but can't keep up with the growth. I suppose I could change the rotation to a shorter period, or do it based on file size, but I'll be doing a hardware refresh to a bigger, better box in about eight weeks, so the script will be a stopgap measure. And, I'm not concerned about deleting precious log files; they're parsed on a continuous basis for events of interest. Finally, rather than giving this man a fish, you've helped teach him how to fish, thereby feeding him for live.

My gratitude.

Diggy

Heh, well, as it appears from the OP's explanation the logical thing to do isn't always the right thing to do...

Good to know it's of use.


As we're scripting away here's another take on things zapping everything (or so I'd hope) bigger than half a GB in /var/log:
this may or may not be as you like it and it's not tested so YMMV(VM).

* Whatever you do you should 'kill -HUP $PID' afterwards to allow processes to re-open file descriptors.

unSpawn,

As an aside, the box that this script will be running on has provided great service to us for nearly seven years. As mentioned, it'll be peacefully retired in about two months. I've tried to take all of the necessary precautions in case it fails but, God willing, it'll make it to the finish line.

The last construct you showed me is interesting, and well beyond my scripting level. I'll study it, and test it on a throw-away VM to see how it performs.

Many thanks.

Diggy

A proper burial would be a bit over the top but I hope you give its remnants a good final destination?..


It's simple: run lsof and only show items in /var/log | awk for any line that has "log/" in it and print the last field (file name) | sort output showing unique items only | while read the output line by line into the variable "LOG"; and for each item do
[ `stat the item only returning its size in bytes` and check if the size is greater than $[10240*50] ] if it is then empty the item
done.