questions regarding executing remote scripts

MarcusWebb1966 · 06-02-2011, 09:26 AM

Hi

Ok, so the plan is this: We have a central server for monitoring all kinds of stuff. One of the jobs I've been asked to get it to do is to analyse (albeit in a less than analytical manner) some log files and email the results back. I get that you can :

Code:

ssh $SERVER 'bash -s' < codeyouwanttorun.sh

to execute the code on the remote machine. What I'm kind of hoping is that you can follow my pseudocode and point me in the direction of either the exact code I need or some reading material that will deepen/widen/broaden my knowledge so that I can kludge it together and appear kluge!

Pseudo code follows:

Code:

Read list of servers to check
while not at end of file
   open connection to server
   search for log files and write results to file (find -name -fprint)
   Read list of files to check
   while not at end of file
      get log file name
      grep for FATAL in log file > result
      if not empty file
        copy result file back to main server (scp (?))
        at main server email result to someone ( mailx )
      else
        at main server email "No Errors Today" to someone ( mailx )
      end if
   on to next file
   close connection to server
on to next server
finished

Just let me know what's rational, what's within the realms of the possible and what's completely insane!

For example:
can you ssh from server A onto server B and execute a piece of code that tells server B to ssh back to server A to run mailx? (or is that not only stupid, but reprehensible?)

Thanks in advance!

carltm · 06-03-2011, 07:42 AM

Everything you suggest sounds right. The only thing you might not
be aware of is that you don't need to transfer the data if you're
using an ssh connection. The standard output from the remote server
will be directed to your local host.

So if you want to check multiple files and have the output go into
separate files, you'll have to spawn a new ssh connection each time.
For example:

Code:

for Server in Server1 Server2 Server3; do
  for Log in /var/log/log1 /var/log/log2 /var/log/log3; do
    LocalFile="$Server-`basename $Log`"
    ssh myuser@$Server "grep FATAL $Log 2>&1" > $LocalFile
    if grep FATAL $LocalFile >/dev/null; then
       echo "there was a problem"
       #write a command to send email and/or alert someone
    fi
  done
done

carltm · 06-03-2011, 07:50 AM

Quote:

Originally Posted by MarcusWebb1966

\can you ssh from server A onto server B and execute a piece of code that tells server B to ssh back to server A to run mailx? (or is that not only stupid, but reprehensible?)

It's possible. And I've learned that it's typical to have many ways to
do something in Linux. I almost never say not to do something a certain
way, because there may be a reason to select a more complex solution over
a simple solution for a specific environment.

So it's not stupid or reprehensible, it's just more complex than running
the code on B in an ssh connection and using its output on A to then run
mailx on A.

If you're wondering why I like simple solutions, they are less likely
to break and they are easier to troubleshoot. Not to mention that it
makes life easier 6 months from now when you need to modify it and you
really don't recall what you did in the first place!

MarcusWebb1966 · 06-03-2011, 09:48 AM

Thank you for at least concluding I'm rational, carltm. So I guess I am probably on the right track with this:

Code:

NAMES=( bloat bubbles callisto pandora lapetus tethys dactyl janus metis tad ganymede sponde deimos bab puck)
TOTAL=${#NAMES[@]}

n=1
while [[ $n -le "$TOTAL" ]];
do
        SERVER=${NAMES[$n-1]}
        echo "Checking server :   $SERVER"
        echo ""
        ssh $SERVER 'bash -s' < checksinthemail.sh
        echo ""
        echo "---"
        echo ""
        let n++
done
echo "Finished!"

checksinthemail.sh

Code:

#!/bin/sh
        echo "Checking server :   $SERVER"
        echo ""
#
#       Find errors in PHP logs which should be in either of the places below
#

        find /var/log -name 'php_errors.log' -fprint '/tmp/var_php.txt'
        find /home/dev/apache -name 'php_errors.log -fprint '/tmp/homedev_php.txt'
        cat /tmp/var_php.txt >> /tmp/homedev_php.txt
#
#       And if the combined files are zero bytes then there were no errors...
#
        if [[ stat -c %s '/tmp/homedev_php.txt' < 0 ]]
                touch /tmp/result.txt
                echo "No errors on server \$SERVER today - well done!" > /tmp/result.txt
        else
#
#       deal with the results...
#
                touch /tmp/result.txt
                echo 'PHP Errors on Server \$SERVER' > /tmp/result.txt
                echo '==============================================' >> /tmp/result.txt
                echo '' >> /tmp/result.txt
                linecount=0
                cat /tmp/homedev_php.txt | while read linefromfile ;
                do
                linecount=$((linecount+1))
                echo "$linecount:  $linefromfile"
                echo "$linefromfile" >> /tmp/result.txt
                echo "----------------------------------------------------------" >> /tmp/result.txt
                # cat $linefromfile | sed -n '/PHP Fatal error/,/^\[/p'
                cat $linefromfile | sed -n '/PHP Fatal error/,/^\[/p' >> /tmp/result.txt
                echo "" >> /tmp/result.txt
                echo "" >> /tmp/result.txt
                echo ""
                done
                echo "done!"
        fi

#
#       Now send the results back to...
#
        scp  /tmp/result.txt peach.cwtonline.co.uk:result.txt
#
#       clean up after yourself
#
#       rm /tmp/result.txt,/tmp/*php.txt -q > /dev/null

citm.sh

Code:

#!/bin/sh
#       deal with the results...
#
                touch /tmp/result.txt
                echo 'PHP Errors on Server \$SERVER' > /tmp/result.txt
                echo '==============================================' >> /tmp/result.txt
                echo '' >> /tmp/result.txt
                linecount=0
                cat /tmp/homedev_php.txt | while read linefromfile
                do
                linecount=$((linecount+1))
                echo "$linecount:  $linefromfile"
                echo "$linefromfile" >> /tmp/result.txt
                echo "----------------------------------------------------------" >> /tmp/result.txt
                cat $linefromfile | sed -n '/PHP Fatal error/,/^\[/p' >> /tmp/result.txt
                echo "" >> /tmp/result.txt
                echo "" >> /tmp/result.txt
                echo ""
                done
                echo "done!"
#
#       Now email the results back to...
#
        mailx -s "\$SERVER PHP Error Report" anonymous@nodomain.com  < /tmp/result.txt
#
#       clean up after yourself
#
#       rm /tmp/result.txt,/tmp/*php.txt -q > /dev/null

Now here's the bizarre part of all of this: citm is a cut down version of checksinthemail.sh.

citm.sh works - checksinthemail.sh doesn't.

The error eppears to be in the line with sed in it but for the life of me I am utterly stumped as to why it fails and the other doesn't. from the little I've worked out I think it may be to do with ssh as a job rather than a terminal (although I have tried adding the -t qualifier to ssh in the original code. I'll look at your code in more detail and try adapting it to what I think I ought to be getting and in the community spirit, will let all have access to the finished article.

MarcusWebb1966 · 08-22-2011, 09:18 AM

Well, we're a few weeks down the line now and not much has happened evolutionarily speaking. Now, however, I've run into a very odd thing. Whenever I run these scripts interactively all seems peachy and the servers do as expected, (i.e. a job is executed on the remote server that I need it to.

However, when this runs as a cron job, no output is returned from the remotely executed script, and for the life of me I cannnot figure out why.

Before you ask me:
- I'm using key-pair authentication and it works fine.
- other cron jobs that call our remote servers seem to run fine but everything that they are being asked to achieve is all put on one line separated with pipes, and as good as I may be with pipes, I'm not sure that what I'm asking this job to do is up to being done as a single command string separated by piping!

Your thoughts and ideas gratefully received as ever!