LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 11-15-2007, 05:41 AM   #1
helptonewbie
Member
 
Registered: Aug 2006
Location: England Somewhere
Distribution: Mandriva, PCLinuxOS, Karoshi, Suse, Redhat, Ubuntu
Posts: 518

Rep: Reputation: 39
Question Script to read a log file??


Hello,
I've got a program that creates its own log file in normal text file, what i would like to be able to do is create another bash script that could perhaps sort through the log file and pick out important lines of it depending on the test string contained on the line. Rather like how swatch works but i'd like my own independent script. Its not something that will run constantly or on a loop just something that the user can run every now and then and it pulls in the current log file and re-creates it only showing specific strings i've listed in another file for example.

does anyone know of a script already out there that could customise to my own needs, or will i need to start from scratch, probably using grep's etc etc.

Tanks for any guidance
 
Old 11-15-2007, 05:50 AM   #2
hairysocks
Member
 
Registered: May 2002
Location: Thorverton, Near Exeter, Devon, England
Distribution: Ubuntu 10.04 (used to be Red Hat 7.1, then Red Hat 9, then FC 2, FC 5, FC 6, FC 9 and Ubuntu 8.04)
Posts: 105

Rep: Reputation: 16
Use grep to search the file for specific strings.
 
Old 11-15-2007, 06:10 AM   #3
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 21,132

Rep: Reputation: 4121Reputation: 4121Reputation: 4121Reputation: 4121Reputation: 4121Reputation: 4121Reputation: 4121Reputation: 4121Reputation: 4121Reputation: 4121Reputation: 4121
Personally I'd use perl - maybe have a look at logwatch.
 
Old 11-15-2007, 06:20 AM   #4
PAix
Member
 
Registered: Jul 2007
Location: United Kingdom, W Mids
Distribution: SUSE 11.0 as of Nov 2008
Posts: 195

Rep: Reputation: 40
Sounds like a job for Bash and Awk. If you have interest in a number of log files (1 or more), identify them in a selection file and in a loop have Awk create/append to your own chosen extract file. Awk is a super choice for selecting from your logs depending on pattern matches and formatting the output. When your loop of 1 or more files completes, head up and tail the extract with the appropriate email information and pipe the lot into xmail or what-have-you. after which you can either archive or delete your extract file ready for the next run. If you want you could have it run each day or once a week etc using at and just have it arrive in time for reading with your coffee. The possibilities are fairly endless and could save a lot of trawling through logs if you know what it is that you are interested in and if the logs have unique tags to allow particular types of event to be pulled out easily. Happy hacking.

PAix
 
Old 11-15-2007, 08:02 AM   #5
helptonewbie
Member
 
Registered: Aug 2006
Location: England Somewhere
Distribution: Mandriva, PCLinuxOS, Karoshi, Suse, Redhat, Ubuntu
Posts: 518

Original Poster
Rep: Reputation: 39
yep thats all rather along the lines what i was thinking, sorry haven't really taken a look at perl yet (not had the need to), and logwatch i also know of but don't require. But i didn't know of a decent way to script it. I suppose its a grep issue i have, i'd like to read in each line of text output in a text file, and if a string of text on a line is a match of any of the strings of text i have in another file that i'd like the text to be compared against, then for it to consider that as a match for that line of text copy it to another file and then check the next line of text for other matches from the other file, if a match again print line to other file.

Something like / similar to this would be amazing but not sure if grep can handle it, might need the use of sed as well???
 
Old 11-15-2007, 08:08 AM   #6
angrybanana
Member
 
Registered: Oct 2003
Distribution: Archlinux
Posts: 147

Rep: Reputation: 21
Quote:
Originally Posted by helptonewbie View Post
yep thats all rather along the lines what i was thinking, sorry haven't really taken a look at perl yet (not had the need to), and logwatch i also know of but don't require. But i didn't know of a decent way to script it. I suppose its a grep issue i have, i'd like to read in each line of text output in a text file, and if a string of text on a line is a match of any of the strings of text i have in another file that i'd like the text to be compared against, then for it to consider that as a match for that line of text copy it to another file and then check the next line of text for other matches from the other file, if a match again print line to other file.

Something like / similar to this would be amazing but not sure if grep can handle it, might need the use of sed as well???
Can you please provide an example of your input and expected output.
 
Old 11-15-2007, 08:40 AM   #7
helptonewbie
Member
 
Registered: Aug 2006
Location: England Somewhere
Distribution: Mandriva, PCLinuxOS, Karoshi, Suse, Redhat, Ubuntu
Posts: 518

Original Poster
Rep: Reputation: 39
something like log file might contain:
Code:
#filename: log.txt
words and stuff problem found
blah blah un-interesting
blah
blah1
blah2
other words problem found error 2 string appears on line so should be copied to other log file
yep same again another error ahh haa
hey ho what you know blah blah
other log blah
error code 4 more problems
thats an example of log file obvious not the exact one but still you can get the idea.
Now a file of text strings to match against.
Code:
#filename: matchtext.txt
"problem found"
"problem found error 2"
"another error"
"error code 4"
so if any string of text on a line in log.txt matches any of the text strings from matchtext.txt, i'd like to be able to take the line thats matched and copy paste so to speak, probably more a redirect to another output file (another log file) called sivedog.txt

Thanks regards
 
Old 11-15-2007, 09:06 AM   #8
DotHQ
Member
 
Registered: Mar 2006
Location: Ohio, USA
Distribution: Red Hat, Fedora, Knoppix,
Posts: 548

Rep: Reputation: 33
awk sounds like a good way of accomplishing this with a script (easier than sed anyway).
 
Old 11-15-2007, 11:05 AM   #9
helptonewbie
Member
 
Registered: Aug 2006
Location: England Somewhere
Distribution: Mandriva, PCLinuxOS, Karoshi, Suse, Redhat, Ubuntu
Posts: 518

Original Poster
Rep: Reputation: 39
does anyone have a script, or can anyone help me write one or start an initial outline of one for this problem??

Thannks and Regards
 
Old 11-15-2007, 05:00 PM   #10
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Rocky 9.2
Posts: 18,360

Rep: Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751
Code:
# Set IFS to (actual) newline (\n) only
IFS=

for match_str in `cat matchtext.txt`
do
    grep "$match_str" log.txt >> out.log
done
Use nested loops or wildcard for log files if more than one
 
Old 11-16-2007, 03:14 PM   #11
helptonewbie
Member
 
Registered: Aug 2006
Location: England Somewhere
Distribution: Mandriva, PCLinuxOS, Karoshi, Suse, Redhat, Ubuntu
Posts: 518

Original Poster
Rep: Reputation: 39
Thats fantastic thanks!!!!
Worked like a charm
 
Old 11-17-2007, 08:18 PM   #12
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Rocky 9.2
Posts: 18,360

Rep: Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751
any time
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
How to create log file from script fotoguy Programming 12 03-27-2012 06:39 AM
How to read numbers from a log file and to compute an average? 3ttt Programming 4 07-02-2007 03:08 AM
script to read file into array STAGED Programming 4 01-27-2006 08:06 PM
Script to create a log file satimis Programming 15 01-16-2006 06:39 AM
How do you read a log file as it is being updated? anti_trend Linux - Software 2 04-02-2004 06:51 AM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 11:49 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration