Welcome to the most active Linux Forum on the web.
Go Back > Forums > Linux Forums > Linux - Newbie
User Name
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!


  Search this Thread
Old 06-23-2009, 10:57 AM   #1
LQ Newbie
Registered: Jun 2009
Posts: 2

Rep: Reputation: 0
copy last 10000+ lines of large text file to a temporary file


I need to periodically copy the last 81424 lines of a large (> 5GB) text file that is an output of a program to a temp file. The text file is not arranged in columns, but it does have a repetitive structure (repeated every 81424 lines). The program is continually writing new data to the text file. I need to scan this data at very small time intervals (less than every 15 sec.) to stop the program as soon as a certain condition is met in the data.

I have tried the following, but I seem to not be able to tail more than 10000 lines at a time:
tail -n 81424 f.txt > ftemp.txt

I also tried these things but they take too much wall clock time (4min+)

1. counting the number of lines in the file, using `cat f.txt | wc -l` and then using head and tail in a pipeline to print out the last 81424 lines of the file (lines #totallines-81424-1 to #totallines).

2. using sed from the infamous 'sed one-liners' to output the last 81424 lines:
sed -e :a -e '$q;N;81425,$D;ba' f.txt

Any suggestions? Is there a way to change the limit on the number of lines that can be output using tail? Can I copy the last X-sized chunk (in bytes or something) to another file somehow?

Old 06-23-2009, 02:54 PM   #2
LQ Guru
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 6,222
Blog Entries: 9

Rep: Reputation: 835Reputation: 835Reputation: 835Reputation: 835Reputation: 835Reputation: 835Reputation: 835
You might try the split command.

Type "man split" for more information.
1 members found this post helpful.
Old 06-23-2009, 03:10 PM   #3
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,041
Blog Entries: 11

Rep: Reputation: 907Reputation: 907Reputation: 907Reputation: 907Reputation: 907Reputation: 907Reputation: 907Reputation: 907
Or tac & awk?

tac f.txt|awk '{if(NR>81424){nextfile}else{print $0}}'|tac > ftemp
Untested, and NO idea how fast this would be.

Old 06-24-2009, 02:43 PM   #4
LQ Newbie
Registered: Jun 2009
Posts: 2

Original Poster
Rep: Reputation: 0
Originally Posted by Tinkster View Post
Or tac & awk?

tac f.txt|awk '{if(NR>81424){nextfile}else{print $0}}'|tac > ftemp
Untested, and NO idea how fast this would be.



Thanks a lot! This works and is very fast.

Here's what I finally ended up using (this outputs a little less then 3 chunks of my data, where each chunk is separated by the word "time"):


tac $EL|awk '/time/ {++i}{if(i>2){nextfile}else{print $0}}'|tac > $tEL



bash, file, sed, tail

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
Remove lines in a text file based on another text file asiandude Programming 10 01-29-2009 10:59 AM
Adding lines of text to beginning of a text file BillKat Programming 2 01-19-2009 10:40 AM
extracting a chunk of text from a large text file lothario Linux - Software 3 02-28-2007 08:16 AM
Last 10 lines of a large file loopoo Linux - Newbie 2 05-10-2006 03:01 AM
How to copy mysql file into text file lumba General 0 09-26-2005 05:08 AM

All times are GMT -5. The time now is 04:31 PM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration