LinuxQuestions.org - Bash: using file as input for grep/tail and then some..

- Programming (https://www.linuxquestions.org/questions/programming-9/)

- - Bash: using file as input for grep/tail and then some.. (https://www.linuxquestions.org/questions/programming-9/bash-using-file-as-input-for-grep-tail-and-then-some-731432/)

Bash: using file as input for grep/tail and then some..

Hi,

A bit of a background on the script I am trying to make first of all.

I have a mysql database with some users in, I am taking a dump of the user table into a file, then I am using that file as the input to grep against the ftp log. I need to see the last 5 entries from the ftp log based on the user. For example:

UserA:
x.x.x9.x - usera.x.co.uk [31/May/2009:11:50:00 +0100] "PUT /home/uploads/usera/x.txt" 200 77
etc

UserB:
x.x.x9.x - userb.x.co.uk [31/May/2009:11:50:00 +0100] "PUT /home/uploads/userb/x.txt" 200 77
x.x.x9.x - userb.x.co.uk [31/May/2009:11:50:00 +0100] "PUT /home/uploads/userb/x.txt" 200 77
x.x.x9.x - userb.x.co.uk [31/May/2009:11:50:00 +0100] "PUT /home/uploads/userb/x.txt" 200 77
x.x.x9.x - userb.x.co.uk [31/May/2009:11:50:00 +0100] "PUT /home/uploads/userb/x.txt" 200 77

I also need to (somehow) run a query on the ftp log to find if someone hasnt uploaded data in the past 2 weeks. I also on top of this need to count how many times the username appears in the logs. The username is part of the /home/uploads/ also. The logfile rotation will be changed to rotate every 2 weeks with this script being ran about 12 hours before the log roates, but it will be ran on a weekly basis. I then require all the data to be put into one easy to read log for the managing directors to look at. Needless to say I have been thrown in the deep end and have never really used bash to this extent before.

So far I have this, which is very messy but gets half the job done, I seem to have hit a wall (programmers block?) and cannot seem to find my way around it, any nudges in the correct direction would be very very useful.

Code:

#!/usr/bin/env bash



set -e



FTPL='pureftpd.log'

UFILE='users.txt'

TIMESTAMP=$(date +%Y%m%d)



echo "Checking to see if users file exists..."



# if to see if the users file exists. If it does move to to a backup file and name it after the current timestamp



if [ -f users.txt ]

then

        echo "Yes it does, moving to backup file"

        mv users.txt users\_$TIMESTAMP.txt

else

        echo "no file"        

fi



# here we connect to the mySQL database and pull a list of the current users down and put them into a text file.        

echo " connecting to mysql to get users"

sleep 5        

# I know I shouldnt be using the PW here but I have been told to :/

mysql -h db.xxx.xxx.xx -u xxxxxx --password=xxxxxxx -e "use FTP; SELECT SUBSTRING_INDEX(user , '.', 1) AS username from users;" > users.txt

echo "done!"

echo "now what we need to do is read the user file, then grep the ftp log against that."

sleep 5 

while read line; 

do 

        echo "Now trying to read the user list"

        echo "${line}";

        echo "reading the file.."

        echo "grepping for the users"

        grep "$line" pureftpd.log | tail -n 5 >> main.log

done < <(cat $UFILE)        

sleep 5

echo "now we are going to check for people who have NOT uploaded"

while read line;

if [ -f notuploaded.txt ]

then

        echo "Yes it does, moving to backup file"

        mv notuploaded.txt noupload\_$TIMESTAMP.txt

else

        echo "no sign of it, creating new file"        

fi

do

        echo "$line";

        grep -v "$line" pureftpd.log > notuploaded.log

done < <(cat $UFILE) 



sleep 5

Any slight nudges or suggestions on what to do would earn my eternal gratitude!

Thank you!

Re: Bash: using file as input for grep/tail and then some..

I don't know about eternal gratitude, but I have never gotten one before, so it would be nice to get one just for the experience :-) Just kidding.

Here is a script:

Code:

system mysql -h db.xxx.xxx.xx -u xxxxxx --password=xxxxxxx -e "use FTP; SELECT SUBSTRING_INDEX(user , '.', 1) AS username from users;" > users.txt

# Read users one by one

var str users, user ; cat "users.txt" > $users

lex "1" $users > $user

while ($user <> "")

do

    # Get the next user for next time

    lex "1" $users > $user



    # Get all lines containing this user from pureftpd.log .

    var str log ; script SS_FindStr.txt files("pureftpd.log") str($user) > $log



    # Get the number of lines in $log

    var int lines ; set $lines = { len $log }



    # We want to remove $lines-5 lines from $log.

    lex (makestr(int($lines-5))+"]") $log > null



    # Last 5 lines for user $user are in $log. Print.

    echo $log

done

It's written in biterscripting. Download it free from http://www.biterscripting.com . I am using one of their sample scripts SS_FindStr. There are instructions in their installation guide on how to install all their sample scripts. Feel free to translate into bash.

Randi

Moin,

first, the "(cat $UFILE)" is not necessary. The line "done <<$UFILE" should do it.

To get the users, which do not appear in the ftp log, the "-v" option of grep is the wrong way. You'll get all lines NOT belonging to this user.

You can use the -c or the -q options:

Code:

if test `grep -c "$line" $UFILE` -eq 0; then ...

if ! grep -q "$line" $UFILE; then ...

Jan

Quote:

Originally Posted by RandiR (Post 3567049)

I don't know about eternal gratitude, but I have never gotten one before, so it would be nice to get one just for the experience :-) Just kidding.

Here is a script:

Code:

system mysql -h db.xxx.xxx.xx -u xxxxxx --password=xxxxxxx -e "use FTP; SELECT SUBSTRING_INDEX(user , '.', 1) AS username from users;" > users.txt

# Read users one by one

var str users, user ; cat "users.txt" > $users

lex "1" $users > $user

while ($user <> "")

do

    # Get the next user for next time

    lex "1" $users > $user



    # Get all lines containing this user from pureftpd.log .

    var str log ; script SS_FindStr.txt files("pureftpd.log") str($user) > $log



    # Get the number of lines in $log

    var int lines ; set $lines = { len $log }



    # We want to remove $lines-5 lines from $log.

    lex (makestr(int($lines-5))+"]") $log > null



    # Last 5 lines for user $user are in $log. Print.

    echo $log

done

Thanks for that, but I can't install anything on the box. I do have access, it is just...frowned upon, although that does look very nice indeed. I may have to see if it is possible I can put it on. I am (and would prefer) to use bash, but there is also phpcli, perl and python on the box that can be used for this. Again, thank you for the help!

Quote:

Originally Posted by jan61 (Post 3567097)

Code:

if test `grep -c "$line" $UFILE` -eq 0; then ...

if ! grep -q "$line" $UFILE; then ...

Jan

Thank you very much for the pointers. I will go back and look over what I have done and see if I can squeeze in what you have suggested..

Thank you

Quote:

Needless to say I have been thrown in the deep end

lol, that's the funniest thing i've read today, sorry for not being able to help