LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Enterprise Linux Forums > Linux - Enterprise
User Name
Password
Linux - Enterprise This forum is for all items relating to using Linux in the Enterprise.

Notices


Reply
  Search this Thread
Old 03-11-2014, 10:59 AM   #1
RHCE_ran
Member
 
Registered: Oct 2013
Posts: 90

Rep: Reputation: Disabled
Best way to extract lines from a log file in Linux


My query is that I have a log file which is around 7 GB and I want to extract lines between a certain time range which should total up to half of the size to 3.5 GB. The lines start with a time range like "2014-03-11 17:35:00". I want to extract the lines between "2014-03-11 17:35:00" to "2014-03-11 18:05:00". What should be the best way as there may be a grep command or a sed command to do it.

I hope, my question is clear.

Please revert with the reply to my query.

Regards
 
Old 03-11-2014, 11:51 AM   #2
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 16,275

Rep: Reputation: 5463Reputation: 5463Reputation: 5463Reputation: 5463Reputation: 5463Reputation: 5463Reputation: 5463Reputation: 5463Reputation: 5463Reputation: 5463Reputation: 5463
There is no best solution, but there are a lot of different languages..... What do you prefer? What have you tried so far?
you can use for example awk (but it will look similar in perl too):
Code:
# pseudo code
awk '/first date filter/ { set a switch to 1 }
     /last date filter/ { set that switch to 0 }
     print lines if switch is set to 1
'
hope this helps
 
Old 03-12-2014, 02:22 AM   #3
RHCE_ran
Member
 
Registered: Oct 2013
Posts: 90

Original Poster
Rep: Reputation: Disabled
Thanks for your help. Request you to please explain the command also with special reference to the switch option as in { set a switch to 1 }.

Waiting for your revert.

Regards
 
Old 03-12-2014, 02:29 AM   #4
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 16,275

Rep: Reputation: 5463Reputation: 5463Reputation: 5463Reputation: 5463Reputation: 5463Reputation: 5463Reputation: 5463Reputation: 5463Reputation: 5463Reputation: 5463Reputation: 5463
that was not a script, but pseudo-code. It means you can implement your script based on that information, by translating the written text to commands. That "switch" is usually a variable....
 
Old 03-12-2014, 04:57 AM   #5
descendant_command
Senior Member
 
Registered: Mar 2012
Posts: 1,767

Rep: Reputation: 578Reputation: 578Reputation: 578Reputation: 578Reputation: 578Reputation: 578
TBH I'd just do it with a series of simple greps appending to a file and repeat as necessary.
http://xkcd.com/1319/
 
Old 03-12-2014, 06:03 AM   #6
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 7.7 (?), Centos 8.1
Posts: 17,849

Rep: Reputation: 2584Reputation: 2584Reputation: 2584Reputation: 2584Reputation: 2584Reputation: 2584Reputation: 2584Reputation: 2584Reputation: 2584Reputation: 2584Reputation: 2584
If you can't guarantee that the exact datetime stamps you want will actually exist in the logfile, then I'd use eg Perl and write a program that eg converted each datetime stamp to Epoch seconds and compared to your desired datetimes.

OTOH, for exact matches, sed can do it
Code:
sed -n '/2014-03-11 17:35:00/,/2014-03-11 18:05:00/p' yourfile
 
Old 03-12-2014, 06:13 AM   #7
cliffordw
Member
 
Registered: Jan 2012
Location: South Africa
Posts: 506

Rep: Reputation: 194Reputation: 194
Quote:
Originally Posted by pan64 View Post
There is no best solution, but there are a lot of different languages..... What do you prefer? What have you tried so far?
you can use for example awk (but it will look similar in perl too):
Code:
# pseudo code
awk '/first date filter/ { set a switch to 1 }
     /last date filter/ { set that switch to 0 }
     print lines if switch is set to 1
'
hope this helps
Hi,

The above approach will only work if both the 1st & last dates are found. What if those exact times (either/both) have no entries in the log, but there are entries between these times?

Grep for a pattern (or six) might be easier if this is a once off requirement, or the start & end times always follow the same rules, but could get messy if not.

Personally I'd write a bit of PERL code to parse the timestamps into seconds (using the Time::Local module or similar) & then use that numeric timestamp to filter. May be overkill, though :-)
 
Old 03-12-2014, 07:45 AM   #8
allend
LQ 5k Club
 
Registered: Oct 2003
Location: Melbourne
Distribution: Slackware-current
Posts: 5,596

Rep: Reputation: 2180Reputation: 2180Reputation: 2180Reputation: 2180Reputation: 2180Reputation: 2180Reputation: 2180Reputation: 2180Reputation: 2180Reputation: 2180Reputation: 2180
If log.txt contains
Quote:
2014-03-11 17:34:00 line0
2014-03-11 17:35:00 line1
2014-03-11 17:36:00 line2
2014-03-11 17:37:00 line3
2014-03-11 17:38:00 line4
2014-03-11 17:53:00 line5
2014-03-11 18:05:00 line6
2014-03-11 18:06:00 line7
then
Code:
awk '{split($1,d,"-");split($2,t,":");e=mktime(d[1]" "d[2]" "d[3]" "t[1]" "t[2]" "t[3]);if (e>=1394519700 && e<=1394521500) print $0}' log.txt
produces
Quote:
2014-03-11 17:35:00 line1
2014-03-11 17:36:00 line2
2014-03-11 17:37:00 line3
2014-03-11 17:38:00 line4
2014-03-11 17:53:00 line5
2014-03-11 18:05:00 line6
 
1 members found this post helpful.
Old 03-12-2014, 11:43 PM   #9
cliffordw
Member
 
Registered: Jan 2012
Location: South Africa
Posts: 506

Rep: Reputation: 194Reputation: 194
Quote:
Originally Posted by allend View Post
If log.txt contains
Code:
awk '{split($1,d,"-");split($2,t,":");e=mktime(d[1]" "d[2]" "d[3]" "t[1]" "t[2]" "t[3]);if (e>=1394519700 && e<=1394521500) print $0}' log.txt
produces
Beautiful! Didn't know mktime is available in awk. This is much shorter than the perl equivalent I suggested :-)
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
extract lines from log file using different parameters veda92 Programming 9 07-07-2013 11:44 PM
[SOLVED] Extract a Sequence of lines from a Text File sparker1970 Linux - General 2 04-04-2012 09:56 PM
How to extract lines from file using AWK keenboy Linux - General 7 08-05-2010 08:29 AM
Extract uncommented lines from a file aarav2306 Linux - Newbie 6 07-11-2009 11:27 AM
how to extract certain lines from a log file Avatar Linux - Newbie 3 02-11-2005 09:51 AM

LinuxQuestions.org > Forums > Enterprise Linux Forums > Linux - Enterprise

All times are GMT -5. The time now is 04:19 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration