LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 06-08-2009, 10:12 AM   #1
sleipnir
LQ Newbie
 
Registered: Jun 2009
Posts: 4

Rep: Reputation: 0
Bash: using file as input for grep/tail and then some..


Hi,

A bit of a background on the script I am trying to make first of all.

I have a mysql database with some users in, I am taking a dump of the user table into a file, then I am using that file as the input to grep against the ftp log. I need to see the last 5 entries from the ftp log based on the user. For example:

UserA:
x.x.x9.x - usera.x.co.uk [31/May/2009:11:50:00 +0100] "PUT /home/uploads/usera/x.txt" 200 77
etc

UserB:
x.x.x9.x - userb.x.co.uk [31/May/2009:11:50:00 +0100] "PUT /home/uploads/userb/x.txt" 200 77
x.x.x9.x - userb.x.co.uk [31/May/2009:11:50:00 +0100] "PUT /home/uploads/userb/x.txt" 200 77
x.x.x9.x - userb.x.co.uk [31/May/2009:11:50:00 +0100] "PUT /home/uploads/userb/x.txt" 200 77
x.x.x9.x - userb.x.co.uk [31/May/2009:11:50:00 +0100] "PUT /home/uploads/userb/x.txt" 200 77

I also need to (somehow) run a query on the ftp log to find if someone hasnt uploaded data in the past 2 weeks. I also on top of this need to count how many times the username appears in the logs. The username is part of the /home/uploads/ also. The logfile rotation will be changed to rotate every 2 weeks with this script being ran about 12 hours before the log roates, but it will be ran on a weekly basis. I then require all the data to be put into one easy to read log for the managing directors to look at. Needless to say I have been thrown in the deep end and have never really used bash to this extent before.

So far I have this, which is very messy but gets half the job done, I seem to have hit a wall (programmers block?) and cannot seem to find my way around it, any nudges in the correct direction would be very very useful.

Code:
#!/usr/bin/env bash

set -e

FTPL='pureftpd.log'
UFILE='users.txt'
TIMESTAMP=$(date +%Y%m%d)

echo "Checking to see if users file exists..."

# if to see if the users file exists. If it does move to to a backup file and name it after the current timestamp

if [ -f users.txt ]
then
	echo "Yes it does, moving to backup file"
	mv users.txt users\_$TIMESTAMP.txt
else
	echo "no file"	
fi

# here we connect to the mySQL database and pull a list of the current users down and put them into a text file. 	
echo " connecting to mysql to get users"
sleep 5	
# I know I shouldnt be using the PW here but I have been told to :/
mysql -h db.xxx.xxx.xx -u xxxxxx --password=xxxxxxx -e "use FTP; SELECT SUBSTRING_INDEX(user , '.', 1) AS username from users;" > users.txt
echo "done!"
echo "now what we need to do is read the user file, then grep the ftp log against that."
sleep 5 
while read line; 
do 
	echo "Now trying to read the user list"
	echo "${line}";
	echo "reading the file.."
	echo "grepping for the users"
	grep "$line" pureftpd.log | tail -n 5 >> main.log
done < <(cat $UFILE)	
sleep 5
echo "now we are going to check for people who have NOT uploaded"
while read line;
if [ -f notuploaded.txt ]
then
	echo "Yes it does, moving to backup file"
	mv notuploaded.txt noupload\_$TIMESTAMP.txt
else
	echo "no sign of it, creating new file"	
fi
do
	echo "$line";
	grep -v "$line" pureftpd.log > notuploaded.log
done < <(cat $UFILE) 

sleep 5

Any slight nudges or suggestions on what to do would earn my eternal gratitude!

Thank you!
 
Old 06-08-2009, 12:50 PM   #2
RandiR
LQ Newbie
 
Registered: Mar 2009
Posts: 6

Rep: Reputation: 1
Re: Bash: using file as input for grep/tail and then some..

I don't know about eternal gratitude, but I have never gotten one before, so it would be nice to get one just for the experience :-) Just kidding.

Here is a script:

Code:
system mysql -h db.xxx.xxx.xx -u xxxxxx --password=xxxxxxx -e "use FTP; SELECT SUBSTRING_INDEX(user , '.', 1) AS username from users;" > users.txt
# Read users one by one
var str users, user ; cat "users.txt" > $users
lex "1" $users > $user
while ($user <> "")
do
    # Get the next user for next time
    lex "1" $users > $user

    # Get all lines containing this user from pureftpd.log .
    var str log ; script SS_FindStr.txt files("pureftpd.log") str($user) > $log

    # Get the number of lines in $log
    var int lines ; set $lines = { len $log }

    # We want to remove $lines-5 lines from $log.
    lex (makestr(int($lines-5))+"]") $log > null

    # Last 5 lines for user $user are in $log. Print.
    echo $log
done
It's written in biterscripting. Download it free from http://www.biterscripting.com . I am using one of their sample scripts SS_FindStr. There are instructions in their installation guide on how to install all their sample scripts. Feel free to translate into bash.

Randi
 
Old 06-08-2009, 01:41 PM   #3
jan61
Member
 
Registered: Jun 2008
Posts: 235

Rep: Reputation: 47
Moin,

first, the "(cat $UFILE)" is not necessary. The line "done <<$UFILE" should do it.

To get the users, which do not appear in the ftp log, the "-v" option of grep is the wrong way. You'll get all lines NOT belonging to this user.

You can use the -c or the -q options:
Code:
if test `grep -c "$line" $UFILE` -eq 0; then ...
if ! grep -q "$line" $UFILE; then ...
Jan
 
Old 06-09-2009, 02:03 AM   #4
sleipnir
LQ Newbie
 
Registered: Jun 2009
Posts: 4

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by RandiR View Post
I don't know about eternal gratitude, but I have never gotten one before, so it would be nice to get one just for the experience :-) Just kidding.

Here is a script:

Code:
system mysql -h db.xxx.xxx.xx -u xxxxxx --password=xxxxxxx -e "use FTP; SELECT SUBSTRING_INDEX(user , '.', 1) AS username from users;" > users.txt
# Read users one by one
var str users, user ; cat "users.txt" > $users
lex "1" $users > $user
while ($user <> "")
do
    # Get the next user for next time
    lex "1" $users > $user

    # Get all lines containing this user from pureftpd.log .
    var str log ; script SS_FindStr.txt files("pureftpd.log") str($user) > $log

    # Get the number of lines in $log
    var int lines ; set $lines = { len $log }

    # We want to remove $lines-5 lines from $log.
    lex (makestr(int($lines-5))+"]") $log > null

    # Last 5 lines for user $user are in $log. Print.
    echo $log
done
It's written in biterscripting. Download it free from http://www.biterscripting.com . I am using one of their sample scripts SS_FindStr. There are instructions in their installation guide on how to install all their sample scripts. Feel free to translate into bash.

Randi
Thanks for that, but I can't install anything on the box. I do have access, it is just...frowned upon, although that does look very nice indeed. I may have to see if it is possible I can put it on. I am (and would prefer) to use bash, but there is also phpcli, perl and python on the box that can be used for this. Again, thank you for the help!
 
Old 06-09-2009, 02:03 AM   #5
sleipnir
LQ Newbie
 
Registered: Jun 2009
Posts: 4

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by jan61 View Post
Moin,

first, the "(cat $UFILE)" is not necessary. The line "done <<$UFILE" should do it.

To get the users, which do not appear in the ftp log, the "-v" option of grep is the wrong way. You'll get all lines NOT belonging to this user.

You can use the -c or the -q options:
Code:
if test `grep -c "$line" $UFILE` -eq 0; then ...
if ! grep -q "$line" $UFILE; then ...
Jan
Thank you very much for the pointers. I will go back and look over what I have done and see if I can squeeze in what you have suggested..

Thank you
 
Old 06-18-2009, 04:51 PM   #6
kike_coello
Member
 
Registered: Jul 2005
Location: maryland
Distribution: Ubuntu 9.04
Posts: 88

Rep: Reputation: 17
Quote:
Needless to say I have been thrown in the deep end
lol, that's the funniest thing i've read today, sorry for not being able to help
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Trying to understand pipes - Can't pipe output from tail -f to grep then grep again lostjohnny Linux - Newbie 15 03-12-2009 10:31 PM
shell script having multiple grep statements-I want input file to be read only once mukta9003 Linux - Newbie 4 08-27-2008 12:58 AM
[SOLVED] tail and grep issue Wim Sturkenboom Linux - General 6 02-06-2008 07:19 AM
how to use grep and tail to get portion of a file laurasong Linux - General 7 08-19-2007 11:59 PM
Bash: Script with tail and grep Primsi Linux - General 2 11-16-2006 05:30 AM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 01:36 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration