LinuxQuestions.org
Go Job Hunting at the LQ Job Marketplace
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 04-28-2013, 11:32 AM   #1
d.vanheeckeren
LQ Newbie
 
Registered: Apr 2013
Posts: 3

Rep: Reputation: Disabled
Stripping all but lines matching string from text files


Have a question...about Linux...so figured this site was a good place to ask.

I have a bunch of server log files, and I don't need to keep all the information in them, only certain lines.

For example, I want to keep ONLY lines with the keywords "Chat", "Global", "Execute", and "Broadcast". I need the entire line it's on, with date and time, but don't need all the other lines.

I did see another thread stripping out lines by keyword, but I want to do the opposite, stripping out everything BUT those lines.

Thanks in advance for any help!

[Edit] Oh yeah...I should have mentioned that this is for a shell script I want to make to take log files from a samba network share, strip out everything but those lines, and save them to a directory on the web server to allow public viewing of events in the logs (Script will be cron job on web server).

Last edited by d.vanheeckeren; 04-28-2013 at 11:42 AM.
 
Old 04-28-2013, 11:54 AM   #2
TB0ne
Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 14,946

Rep: Reputation: 2671Reputation: 2671Reputation: 2671Reputation: 2671Reputation: 2671Reputation: 2671Reputation: 2671Reputation: 2671Reputation: 2671Reputation: 2671Reputation: 2671
Quote:
Originally Posted by d.vanheeckeren View Post
Have a question...about Linux...so figured this site was a good place to ask.

I have a bunch of server log files, and I don't need to keep all the information in them, only certain lines. For example, I want to keep ONLY lines with the keywords "Chat", "Global", "Execute", and "Broadcast". I need the entire line it's on, with date and time, but don't need all the other lines.

I did see another thread stripping out lines by keyword, but I want to do the opposite, stripping out everything BUT those lines.

[Edit] Oh yeah...I should have mentioned that this is for a shell script I want to make to take log files from a samba network share, strip out everything but those lines, and save them to a directory on the web server to allow public viewing of events in the logs (Script will be cron job on web server).
We'll be glad to help you get a shell script going...so post what you've written/done so far, and where you're stuck. But we aren't going to write it for you.

If it was me, I'd just throw the log file through grep for whatever keywords you want, and output them to another file. Lots of information on the grep man page, and easily found on Google too:
Code:
grep 'Global\|Chat\|Execute\|Broadcast' /path/to/log.file > /path/to/output.file
 
1 members found this post helpful.
Old 04-28-2013, 12:26 PM   #3
d.vanheeckeren
LQ Newbie
 
Registered: Apr 2013
Posts: 3

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by TB0ne View Post
But we aren't going to write it for you.
That's cool, I wasn't expecting that. I removed the pathnames, and I used the three question marks as a placeholder for the actual operation. I thought it should be really simple, so what I've got so far is this:
Code:
mv [logpath]/*.log [destinationpath]
for i in [destinationpath]/*.log; do ???/$i.txt; done
So I will try this now (but using cp instead of mv while testing]:
Code:
mv [logpath]/*.log [destinationpath]
for i in [destinationpath]/*.log; do grep 'Global\|Chat\|execute\|Broadcast' [destinationpath]/$i.log > $i.txt; done
rm *.log
Quote:
Originally Posted by TB0ne View Post
If it was me, I'd just throw the log file through grep for whatever keywords you want, and output them to another file. Lots of information on the grep man page, and easily found on Google too:
Code:
grep 'Global\|Chat\|Execute\|Broadcast' /path/to/log.file > /path/to/output.file
I think the grep is exactly what I was looking for! Simple and effective...sorry, I'm kind of a newbie to linux, and didn't realize that grep could do multiple keywords. I was thinking I was going to have to do another loop for each keyword, but couldn't think of how to do that while still keeping the lines in chronological order. I should have looked though, sorry about that. I'll try it when I get back this evening, and if I run into any problems, I'll consult the manpages first. And thank you, TBOne, for being willing to help!

Last edited by d.vanheeckeren; 04-28-2013 at 12:27 PM.
 
Old 04-28-2013, 12:57 PM   #4
TB0ne
Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 14,946

Rep: Reputation: 2671Reputation: 2671Reputation: 2671Reputation: 2671Reputation: 2671Reputation: 2671Reputation: 2671Reputation: 2671Reputation: 2671Reputation: 2671Reputation: 2671
Quote:
Originally Posted by d.vanheeckeren View Post
That's cool, I wasn't expecting that. I removed the pathnames, and I used the three question marks as a placeholder for the actual operation. I thought it should be really simple, so what I've got so far is this:
Code:
mv [logpath]/*.log [destinationpath]
for i in [destinationpath]/*.log; do ???/$i.txt; done
So I will try this now (but using cp instead of mv while testing]:
Code:
mv [logpath]/*.log [destinationpath]
for i in [destinationpath]/*.log; do grep 'Global\|Chat\|execute\|Broadcast' [destinationpath]/$i.log > $i.txt; done
rm *.log
You would be AMAZED at how many times folks DO expect a script written for them here.
Quote:
I think the grep is exactly what I was looking for! Simple and effective...sorry, I'm kind of a newbie to linux, and didn't realize that grep could do multiple keywords. I was thinking I was going to have to do another loop for each keyword, but couldn't think of how to do that while still keeping the lines in chronological order. I should have looked though, sorry about that. I'll try it when I get back this evening, and if I run into any problems, I'll consult the manpages first. And thank you, TBOne, for being willing to help!
No sweat, and was glad to help. This may fit your needs for now, but regular-expressions are complicated, and VERY powerful. A lot of Linux commands (like grep), can accept them, but getting them formatted correctly can be a chore at times. There are lots of pages that explain regex...my eyes glaze over after looking at them too long, though....

You shouldn't have to move your logs to another location (unless you WANT to), to get them read. You can just tell grep to go through them in place, and output to a different location for display. But if you're wanting to zero-out the log files so you only get new information, you have options:
  • The tail -f command. That will grab any new lines going IN to the log files, check them for the string(s), and hork them out to the output file if they're found.
    Code:
    tail -f /path/to/log.file grep 'Global\|Chat\|Execute\|Broadcast' > /path/to/output.file
  • Swatch or Logwatch: two utilities written specifically to watch log files. They may or may not work for you
 
Old 04-28-2013, 02:56 PM   #5
cortman
Member
 
Registered: Jan 2012
Location: ZZ9 Plural Z Alpha
Distribution: Crunchbang 11, LFS 7.3, DSL 4.1.10, Lubuntu 12.10, Debian 7
Posts: 219

Rep: Reputation: 43
Just as an FYI, I've found O'Reilly's Mastering Regular Expressions to be a really helpful reference- it's written simply and is easy to understand and yet is very in-depth.
 
Old 04-29-2013, 09:00 AM   #6
d.vanheeckeren
LQ Newbie
 
Registered: Apr 2013
Posts: 3

Original Poster
Rep: Reputation: Disabled
Thanks for all the help, guys. I never did get a chance last night to try that script, and today's gonna be really really busy, so I probably can't get to it until tomorrow. TBOne and cortman, thanks for the suggestions, I'm going to see if I can find that book on regular expressions on the amazon ebook store.

And even though I haven't had time to test it yet, I woke up in the middle of the night with another idea. LOL (My weird brain does that random kind of stuff) But anyway, here's my idea:
First, these logs are from a game server, and the original reason I wanted to strip all but the desired lines was so that I could keep a record of what happens, because we're starting a Halo CE and UT99 community, and people are already complaining that an admin has kicked or banned them for no reason, and this way I'd have a record as a way to verify what people claim. I said I wanted to make these viewable publicly also, and I was thinking just text files. But when I woke up in the middle of the night, I realized I'm gonna try making this script create an actual html page (shouldn't be that hard for simple html tags and some basic formatting I would think), and limit the size of each html file to a specific (undetermined so far) size. Then all I'll have to do is create an html page that displays a listing of it's own directory, and the files will be viewable by anybody and there can be no more room for argument. So Wednesday, I'm gonna do my best to get it all done if I have enough time.

On a side note, as these are game server logs from a windows machine (Halo is running on Windows 7, UT server on linux), and not actual linux server log files, I don't know if the swatch or logwatch would help me or not, but I will check into them when I get a chance too.

And again, thanks for the toss-ins!
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
sed or grep : delete lines containing matching text raj000 Linux - General 18 09-08-2012 10:38 AM
[SOLVED] bash matching two files lines by lines rperezalejo Programming 6 01-12-2012 07:42 AM
Stripping lines versus stripping bytes in a bash subshell. poorman_installer Programming 9 10-21-2009 09:36 AM
Rename files by stripping text from either desired string? varney Linux - General 2 02-21-2008 11:04 PM
regular expresions: java: matching lines with out string zymos Programming 1 05-28-2005 10:17 AM


All times are GMT -5. The time now is 11:34 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration