LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 06-23-2009, 10:57 AM   #1
emilyg
LQ Newbie
 
Registered: Jun 2009
Posts: 2

Rep: Reputation: 0
copy last 10000+ lines of large text file to a temporary file


Hello,

I need to periodically copy the last 81424 lines of a large (> 5GB) text file that is an output of a program to a temp file. The text file is not arranged in columns, but it does have a repetitive structure (repeated every 81424 lines). The program is continually writing new data to the text file. I need to scan this data at very small time intervals (less than every 15 sec.) to stop the program as soon as a certain condition is met in the data.

I have tried the following, but I seem to not be able to tail more than 10000 lines at a time:
tail -n 81424 f.txt > ftemp.txt


I also tried these things but they take too much wall clock time (4min+)

1. counting the number of lines in the file, using `cat f.txt | wc -l` and then using head and tail in a pipeline to print out the last 81424 lines of the file (lines #totallines-81424-1 to #totallines).

2. using sed from the infamous 'sed one-liners' to output the last 81424 lines:
sed -e :a -e '$q;N;81425,$D;ba' f.txt


Any suggestions? Is there a way to change the limit on the number of lines that can be output using tail? Can I copy the last X-sized chunk (in bytes or something) to another file somehow?

Thanks!
 
Old 06-23-2009, 02:54 PM   #2
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,831
Blog Entries: 15

Rep: Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669Reputation: 1669
You might try the split command.

Type "man split" for more information.
 
1 members found this post helpful.
Old 06-23-2009, 03:10 PM   #3
Tinkster
Moderator
 
Registered: Apr 2002
Location: earth
Distribution: slackware by choice, others too :} ... android.
Posts: 23,067
Blog Entries: 11

Rep: Reputation: 928Reputation: 928Reputation: 928Reputation: 928Reputation: 928Reputation: 928Reputation: 928Reputation: 928
Or tac & awk?

Code:
tac f.txt|awk '{if(NR>81424){nextfile}else{print $0}}'|tac > ftemp
Untested, and NO idea how fast this would be.



Cheers,
Tink
 
Old 06-24-2009, 02:43 PM   #4
emilyg
LQ Newbie
 
Registered: Jun 2009
Posts: 2

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by Tinkster View Post
Or tac & awk?

Code:
tac f.txt|awk '{if(NR>81424){nextfile}else{print $0}}'|tac > ftemp
Untested, and NO idea how fast this would be.



Cheers,
Tink

Tink,

Thanks a lot! This works and is very fast.

Here's what I finally ended up using (this outputs a little less then 3 chunks of my data, where each chunk is separated by the word "time"):

EL=f.txt
tEL=ftemp

tac $EL|awk '/time/ {++i}{if(i>2){nextfile}else{print $0}}'|tac > $tEL

-Emily
 
  


Reply

Tags
bash, file, sed, tail


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Remove lines in a text file based on another text file asiandude Programming 10 01-29-2009 10:59 AM
Adding lines of text to beginning of a text file BillKat Programming 2 01-19-2009 10:40 AM
extracting a chunk of text from a large text file lothario Linux - Software 3 02-28-2007 08:16 AM
Last 10 lines of a large file loopoo Linux - Newbie 2 05-10-2006 03:01 AM
How to copy mysql file into text file lumba General 0 09-26-2005 05:08 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 06:50 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration