LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices

Reply
 
Search this Thread
Old 02-17-2010, 07:26 AM   #1
nerak99
Member
 
Registered: Feb 2006
Distribution: FC 15
Posts: 42

Rep: Reputation: 2
Filter a large document by line number


I have a 50000 line(ish) set of records in a file. I have another file where I have filtered out all the line numbers for those which have an error of various types. e.g column count, field type etc. I want to get all those lines into a separate file so I can sanitise them. There are abt 3-4000 of them.

How can I access those lines which I want to isolate into a single file?

I have all the usual linux stuff available and a bit of understanding of regexps.
 
Old 02-17-2010, 08:54 AM   #2
nerak99
Member
 
Registered: Feb 2006
Distribution: FC 15
Posts: 42

Original Poster
Rep: Reputation: 2
Grab specific line number from file

After making progress on a problem earlier I now have a file of row number which represent rows that I want to rescue from a file of about 50000 rows.

I want to use the row number file as a filter to determine which of the rows in the 50000 row original file get through to my results file.
 
Old 02-17-2010, 09:19 AM   #3
nerak99
Member
 
Registered: Feb 2006
Distribution: FC 15
Posts: 42

Original Poster
Rep: Reputation: 2
this does what I need

sed -n <line>p <filename>

So a bit of work with emacs on my line numbers file gives stuff like


sed -n 10p allbut1st28.sql
sed -n 24p allbut1st28.sql
sed -n 68p allbut1st28.sql
sed -n 128p allbut1st28.sql
sed -n 134p allbut1st28.sql
sed -n 136p allbut1st28.sql
sed -n 161p allbut1st28.sql
sed -n 162p allbut1st28.sql
sed -n 228p allbut1st28.sql
sed -n 342p allbut1st28.sql
sed -n 412p allbut1st28.sql
sed -n 414p allbut1st28.sql
sed -n 421p allbut1st28.sql
sed -n 510p allbut1st28.sql

Which I then run like allmydudlines.sh >the sql for the dudlines.sql

If I was a guru then I could do the whole thing in a 100 character line with mainly punctuation characters... but then I am not.

At least I don't have to wear sandals in this weather but I could do with the beard.
 
Old 02-17-2010, 10:52 AM   #4
amani
Senior Member
 
Registered: Jul 2006
Location: Kolkata, India
Distribution: 64-bit GNU/Linux, Kubuntu64, Fedora QA, Slackware,
Posts: 2,758

Rep: Reputation: Disabled
is it really beyond http://web.mit.edu/gnu/doc/html/textutils_toc.html ?

If line number does not match value in ___, then append line to file...
 
Old 02-17-2010, 10:57 AM   #5
pixellany
LQ Veteran
 
Registered: Nov 2005
Location: Annapolis, MD
Distribution: Arch/XFCE
Posts: 17,802

Rep: Reputation: 728Reputation: 728Reputation: 728Reputation: 728Reputation: 728Reputation: 728Reputation: 728
Suppose rownumfile contains the row numbers to capture, and rowsfile is to receive the selected lines:

Code:
while read num; do
     sed -n "${num}p" allbut1st28.sql
done < rownumfile > rowsfile
 
Old 02-17-2010, 11:09 AM   #6
pixellany
LQ Veteran
 
Registered: Nov 2005
Location: Annapolis, MD
Distribution: Arch/XFCE
Posts: 17,802

Rep: Reputation: 728Reputation: 728Reputation: 728Reputation: 728Reputation: 728Reputation: 728Reputation: 728
I have merged your two closely-related threads---please keep this all in one place.

thanks
 
  


Reply

Tags
filter, sql


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Printing a Large document on multiple pages eteck Linux - General 3 01-30-2013 08:11 AM
Dansguardian content filter - no progress bar on large downloads __PJ Linux - Software 0 12-08-2009 04:50 AM
Filter through line/s to grab specific fields/data in the line with example shayno90 Linux - Newbie 11 10-14-2009 11:51 AM
ext3 performance -- very large number of files, large filesystems, etc. td3201 Linux - Server 5 11-25-2008 09:28 AM


All times are GMT -5. The time now is 06:36 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration