LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices



Reply
 
Search this Thread
Old 12-14-2006, 02:58 PM   #1
pcorajr
LQ Newbie
 
Registered: Apr 2004
Posts: 14

Rep: Reputation: 0
need help with text manipulation


Hello all I'm a newbie to programming and scripting but so far I been working with a friend on a bash script to automate the download/decryption/text-editing/upload of certain files.

Basically our script downloads a file from a predetermined FTP site, the file is encrypted using PGP so the script also decrypts the file and names it. The script has a lot of code that enables it to send an e-mail wanning to me if it encounters an error (File not been in the ftp site, name of file not matching or file been corrupted etc etc). Anyways its been a fun project so far but now i have hit a wall.

My next step in the project is to do some basic text editing on the file basically i need to delete the first and last line of the file. the file looks something like this:

01030e939-3-20339302039393039403942-12-03920399203
jdkfjei349893894838294
98320498023948928490483489
3924803928409238409388
92304803928402384098320498
9203893r023jf039840390894403984
293-0482390840923840983209
32094802398409238409823039
39480293840923840983204980
3204832948023984023809483029480
932482039840239840392849048
230482309-02398402398402398409238
8320p498230948230948230984230984
4954868

Everyday i will be downloading a file that will have more or less data. but everytimne i have to delete the first and last line, i have read and tried AWK with no luck

anybody can point me in the right direction please.
 
Old 12-14-2006, 03:12 PM   #2
Byenary
Member
 
Registered: Oct 2003
Distribution: edgy dapper feisty
Posts: 33

Rep: Reputation: 15
sed

I think you can do such thin with the sed instruction
 
Old 12-14-2006, 03:20 PM   #3
pcorajr
LQ Newbie
 
Registered: Apr 2004
Posts: 14

Original Poster
Rep: Reputation: 0
Thanks much for pointing out SED i found some stuff on line that looks like is going to help with this:

# delete the first 10 lines of a file
sed '1,10d'

so is this right ?

#Delete the first line of a file
sed '1d'

also to delete last line

sed '$d'

Thanks much.
 
Old 12-14-2006, 04:11 PM   #4
pcorajr
LQ Newbie
 
Registered: Apr 2004
Posts: 14

Original Poster
Rep: Reputation: 0
For some reason when I try to combine sed '1d' and sed '$d' They dont play nice =(
 
Old 12-14-2006, 06:22 PM   #5
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.6, Centos 5.10
Posts: 16,324

Rep: Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041
Well, this suggestion isn't terribly elegant, but if you are using bash, you can use
wc -l
to find how many lines in the file, then subtract 1 to get newlinecnt and use
head, tail to get all but last line & first lines eg if t.t has 9 lines, then
head -8 t.t|tail -7
chops off 1st & last lines.
 
Old 12-14-2006, 08:27 PM   #6
ntubski
Senior Member
 
Registered: Nov 2005
Distribution: Debian
Posts: 2,543

Rep: Reputation: 879Reputation: 879Reputation: 879Reputation: 879Reputation: 879Reputation: 879Reputation: 879
Quote:
Originally Posted by pcorajr
For some reason when I try to combine sed '1d' and sed '$d' They dont play nice =(
How did you combine them? If you did something like
Code:
sed '1d$d'
it won't work because sed will think that is one (malformed) compound command. Try
Code:
sed -e '1d' -e '$d'
works for me.
 
Old 12-14-2006, 08:46 PM   #7
matthewg42
Senior Member
 
Registered: Oct 2003
Location: UK
Distribution: Kubuntu 12.10 (using awesome wm though)
Posts: 3,530

Rep: Reputation: 63
You can either specify multiple commands using multiple -e arguments, or separate commands with a ;
Code:
matthew@chubby:~/tmp$ cat testfile
one Bob a-dealin'
two Bobs a-dealin'
three Bobs a-dealin'
four Bobs a-dealin'
five Bobs a-dealin'
matthew@chubby:~/tmp$ sed -e '1 d' -e '$ d' testfile
two Bobs a-dealin'
three Bobs a-dealin'
four Bobs a-dealin'
matthew@chubby:~/tmp$ sed '1 d ; $ d' testfile
two Bobs a-dealin'
three Bobs a-dealin'
four Bobs a-dealin'
 
Old 12-14-2006, 09:16 PM   #8
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,697
Blog Entries: 5

Rep: Reputation: 241Reputation: 241Reputation: 241
Python alternative:

Code:
data = open("file").readlines()
open("output","w").write(''.join(data[1:-1]))
 
Old 12-14-2006, 09:47 PM   #9
matthewg42
Senior Member
 
Registered: Oct 2003
Location: UK
Distribution: Kubuntu 12.10 (using awesome wm though)
Posts: 3,530

Rep: Reputation: 63
Quote:
Originally Posted by ghostdog74
Python alternative:

Code:
data = open("file").readlines()
open("output","w").write(''.join(data[1:-1]))
It's OK for short files, but if you have a 10 GiG file with millions of lines... not so hot.
 
Old 12-14-2006, 11:35 PM   #10
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,697
Blog Entries: 5

Rep: Reputation: 241Reputation: 241Reputation: 241
Quote:
Originally Posted by matthewg42
It's OK for short files, but if you have a 10 GiG file with millions of lines... not so hot.
true, but there are ways to do it in Python,eg memory map, seek(),tell(), etc even a simple for loop over the lines ...
correct me if i am wrong, but even if sed, awk, grep over 10G file, its slow too right?
 
Old 12-14-2006, 11:46 PM   #11
matthewg42
Senior Member
 
Registered: Oct 2003
Location: UK
Distribution: Kubuntu 12.10 (using awesome wm though)
Posts: 3,530

Rep: Reputation: 63
Quote:
Originally Posted by ghostdog74
true, but there are ways to do it in Python,eg memory map, seek(),tell(), etc even a simple for loop over the lines ...
correct me if i am wrong, but even if sed, awk, grep over 10G file, its slow too right?
Well they'll be slow in that they'll need to read and filter a lot of data, but they'll only keep one line in memory at a time (or a few in the case of a sed program using the N command).

However, I think the approach above with python will try slurp the whole file into memory.
 
Old 12-15-2006, 02:31 AM   #12
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,697
Blog Entries: 5

Rep: Reputation: 241Reputation: 241Reputation: 241
Quote:
Originally Posted by matthewg42
Well they'll be slow in that they'll need to read and filter a lot of data, but they'll only keep one line in memory at a time (or a few in the case of a sed program using the N command).
sure. with a looping in Python, it can achieve "one line in memory at a time"

Quote:
However, I think the approach above with python will try slurp the whole file into memory.
yes, you are right. the alternative assumes the file is small enough to fit into the memory of the system which its running.
 
Old 12-15-2006, 08:33 AM   #13
pcorajr
LQ Newbie
 
Registered: Apr 2004
Posts: 14

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by ntubski
How did you combine them? If you did something like
Code:
sed '1d$d'
it won't work because sed will think that is one (malformed) compound command. Try
Code:
sed -e '1d' -e '$d'
works for me.

Yes that was the mistake i was doing thanks much

matthewg42, thank you for the example it really helped.

Thank you all very much for taking the time to help me out.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Gambas: multi-form text manipulation? LinuxSeeker Programming 3 09-21-2008 07:04 PM
text manipulation in scripts manicman Linux - Newbie 8 02-17-2006 06:04 AM
Manipulation of text files in C++ Hady Programming 5 05-31-2005 09:24 AM
More text manipulation ice_hockey Linux - General 2 05-28-2005 02:43 AM
CLI based Text manipulation selfsck Linux - General 1 07-15-2004 08:11 PM


All times are GMT -5. The time now is 11:49 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration