Help with Rsync and DU commands output to file and mail

ssa2204 · 04-29-2011, 07:12 AM

I have a simple task I run to copy via rsync directories from one machine to another. The command I run for each machine/directory is this:

rsync -vvaz -e ssh root@DS1:/var/log/ /Network/DS/var/log > /Network/DS1/log_1_Monday.txt

or

rsync -vvaz --progress -e ssh root@DS2:/var/etc/ /Network/DS2/var/etc

Afterwards I have this run:
du -h --max-depth=1 /Network/DS1 | tail -n 5 | sort -n > /Network/DS_log.txt

I put them all together and added this to CRON (via Webmin) and it works perfectly. Each night the backup is made, and in the morning a new log is there for the rsync and du output.

Well since I am in the "Linux - Newbie" forum, I suppose you can guess that I have pretty much reached the limits here. As it is I have done a lot of Google searches to get to where I am at right now, but alas Google may be at the end of its ropes in terms of helping me.

My first problem is when I issue a series of commands like:

du -h --max-depth=1 /Network/DS1 | tail -n 5 | sort -n > /Network/DS_log.txt
du -h --max-depth=1 /Network/DS2 | tail -n 5 | sort -n >> /Network/DS_log.txt
du -h --max-depth=1 /Network/DS3 | tail -n 5 | sort -n >> /Network/DS_log.txt

I would like to have each individual output spaced, or separated. So DS1 creates the file DS_log.txt, then DS2's output just adds, ditto DS3. Is there a way to possibly put some output of a command in between to separate in the file?

du -h --max-depth=1 /Network/DS1 | tail -n 5 | sort -n > /Network/DS_log.txt
<SPACE IT>
du -h --max-depth=1 /Network/DS2 | tail -n 5 | sort -n >> /Network/DS_log.txt
<SPACE IT>
du -h --max-depth=1 /Network/DS3 | tail -n 5 | sort -n >> /Network/DS_log.txt
<SPACE IT>
etc.......

Eventually I would like to get the rsync and disk usage commands mailed to me (I work remotely so that it would a real nice addition). I believe this is how I would do it:

du -h --max-depth=1 /Network/DS1 | tail -n 5 | sort -n > /Network/DS_log.txt
cat /Network/DS_log.txt | mail -s"Directory Size" Joe@domain.com

So if I add the "cat /..." at the end of the file it will simply mail that out. OK, now here is my second issue. With the rsync I have 3 machines with around 4 to 5 directories each that I transfer. I want to get the total output (log) of the rsync command sent to me. Problem is if I have each individual rsync command logged to a single file I find that while the simple newb script (if you can call it that) is finished, the file itself is basically empty. If I slowly run each rsync command at a time and give the time to have the file output written then fine. Good lord I hope I am making sense here. Should I have added a warning that this post will probably cause headaches, or fits of laughter

?

tronayne · 04-29-2011, 08:09 AM

OK, here's something you might try.

Let's say you have a shell program to do your logging. Start it with something like this

Code:

#!/bin/sh
#
#     Clear the existing log file
#
> /Network/DS_log.txt
#
#     Make a DU entry in the log (note the 2>&1 -- you get both standard out and standard error that way,
#     also note the >> to append to the previously-emptied log file)
#
du -h --max-depth=1 /Network/DS1 | tail -n 5 | sort -n 2>&1 >> /Network/DS_log.txt
#
#     append a blank line to separate log entries (or put a line of hyphens or whatever)
#
echo "" >> /Network/DS_log.txt
#
#     Do other stuff, like your syncs, adding the 2>&1 and >> as necessary
#
#
#     Mail the log to yourself
#
mail -s "Directory Size" Joe@domain.com < /Network/DS_log.txt

One thing you should probably (always) do if this thing runs from cron is use absolute paths for the executable utilities you're using; e.g., /usr/bin/du rather than just du; the reason for that is that cron doesn't have much of a PATH environment and might not be able to find things you're trying to execute. Be sure to check where your utilities are located. You can find out what cron's PATH is by inserting

Code:

echo ${PATH} >> /Network/DS_log.txt

at the top of the shell program, say right after the line that clears the existing log file.

If your rsync command works manually, it'll also work in a shell program or a cron job -- just make sure to use absolute path, 2>&1 (so you'll see any errors) and >> into the log file.

Hope this helps some.