LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 07-10-2009, 11:15 AM   #1
austin881
LQ Newbie
 
Registered: Jul 2009
Posts: 2

Rep: Reputation: 0
Need help stripping statement from text file, ksh: sed awk?


This is my first time to this site, so go easy on me.

I'm writing a Korn Script that clears out the NO_HW entries from an ioscan. The script takes the output of the ioscan and places into a text file. The text file is reduced to only the lines with "NO_HW" on them.

Example:
Code:
ctl         54  0/3/0/0/0/0.0.21.11.0    sctl           NO_HW       DEVICE       HP      260 SAS AJ940A
ctl         58  0/3/0/0/0/0.0.27.11.0    sctl           NO_HW       DEVICE       HP      270 SAS AJ941A
ctl         59  0/3/0/0/0/0.0.28.9.0     sctl           NO_HW       DEVICE       HP      270 SAS AJ941A
ctl         60  0/3/0/0/0/0.0.29.11.0    sctl           NO_HW       DEVICE       HP      270 SAS AJ941A
ctl         61  0/3/0/0/0/0.0.30.12.0    sctl           NO_HW       DEVICE       HP      270 SAS AJ941A
ctl         62  0/3/0/0/0/0.0.31.12.0    sctl           NO_HW       DEVICE       HP      270 SAS AJ941A
ctl         63  0/3/0/0/0/0.0.32.12.0    sctl           NO_HW       DEVICE       HP      270 SAS AJ941A
I am trying to strip out EVERYTHING except the H/W path.

Example:
Code:
0/3/0/0/0/0.0.21.11.0
So that the text file will look like this:

Code:
0/3/0/0/0/0.0.21.11.0
0/3/0/0/0/0.0.27.11.0
0/3/0/0/0/0.0.28.9.0
0/3/0/0/0/0.0.29.11.0
..and so on. So I can use the rmsf -H 0/3/0/0/0/0.0.21.11.0 command on each H/W Path and remove it.

The command needs to be pretty versatile as the H/W path could be just about anything.

I was thinking this could probably be done with sed or maybe awk but I'm not very proficient in those. Thanks.
 
Old 07-10-2009, 11:32 AM   #2
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950
No need for anything as fancy as sed or awk.

cut -f3 file.txt

This is assuming the fields are separated by tabs. If they're spaces you'll have to add '-d " "' and change the -f number to match the column you want.

Last edited by David the H.; 07-10-2009 at 11:35 AM.
 
Old 07-10-2009, 12:46 PM   #3
David1357
Senior Member
 
Registered: Aug 2007
Location: South Carolina, U.S.A.
Distribution: Ubuntu, Fedora Core, Red Hat, SUSE, Gentoo, DSL, coLinux, uClinux
Posts: 1,302
Blog Entries: 1

Rep: Reputation: 107Reputation: 107
Quote:
Originally Posted by austin881 View Post
I was thinking this could probably be done with sed or maybe awk but I'm not very proficient in those.
Although "David the H." has an easy method, it fails if the column number ever changes. A less fragile approach is
Code:
[user@machine:~]:awk '{ print $3 }' < blah.txt
0/3/0/0/0/0.0.21.11.0
0/3/0/0/0/0.0.27.11.0
0/3/0/0/0/0.0.28.9.0
0/3/0/0/0/0.0.29.11.0
0/3/0/0/0/0.0.30.12.0
0/3/0/0/0/0.0.31.12.0
0/3/0/0/0/0.0.32.12.0
awk is able to handle an arbitrary number of spaces between columns.

Another approach would be to use sed to replace the spaces with commas, and then use cut:
Code:
[user@machine:~]:sed -e 's/  */,/g' < blah.txt | cut -d ',' -f 3
0/3/0/0/0/0.0.21.11.0
0/3/0/0/0/0.0.27.11.0
0/3/0/0/0/0.0.28.9.0
0/3/0/0/0/0.0.29.11.0
0/3/0/0/0/0.0.30.12.0
0/3/0/0/0/0.0.31.12.0
0/3/0/0/0/0.0.32.12.0
NOTE: The sed expression has two spaces before the asterisk. That tells sed to match one or more spaces.
 
Old 07-10-2009, 02:55 PM   #4
jan61
Member
 
Registered: Jun 2008
Posts: 235

Rep: Reputation: 46
Moin,

Quote:
Originally Posted by David1357 View Post
Code:
[user@machine:~]:awk '{ print $3 }' < blah.txt
Why do you use stdin redirect? awk is able to read from files given as command line argument.

Quote:
Originally Posted by David1357 View Post
Code:
[user@machine:~]:sed -e 's/  */,/g' < blah.txt | cut -d ',' -f 3
Again: Why the redirect? The sed is also (like the majority of unix commands) able to retrieve filenames from the command line.

In the second example I'd prefer to let sed do the whole work. No need for cut:
Code:
sed 's/[^ ]* *[^ ]* *\([^ ]*\).*/\1/' file.txt
If you want to catch spaces and tabs:
Code:
sed 's/[^ \t]*[ \t]*[^ \t]*[ \t]*\([^ \t]*\).*/\1/' file.txt
Jan

Last edited by jan61; 07-10-2009 at 02:56 PM.
 
Old 07-13-2009, 10:53 AM   #5
David1357
Senior Member
 
Registered: Aug 2007
Location: South Carolina, U.S.A.
Distribution: Ubuntu, Fedora Core, Red Hat, SUSE, Gentoo, DSL, coLinux, uClinux
Posts: 1,302
Blog Entries: 1

Rep: Reputation: 107Reputation: 107
Quote:
Originally Posted by jan61 View Post
Why do you use stdin redirect? awk is able to read from files given as command line argument.
Habit.

Quote:
Originally Posted by jan61 View Post
Again: Why the redirect? The sed is also (like the majority of unix commands) able to retrieve filenames from the command line.
Again, habit.

Quote:
Originally Posted by jan61 View Post
In the second example I'd prefer to let sed do the whole work. No need for cut:
Code:
sed 's/[^ ]* *[^ ]* *\([^ ]*\).*/\1/' file.txt
Not very readable.

Quote:
Originally Posted by jan61 View Post
If you want to catch spaces and tabs:
Code:
sed 's/[^ \t]*[ \t]*[^ \t]*[ \t]*\([^ \t]*\).*/\1/' file.txt
Even less readable.
 
Old 07-13-2009, 11:11 AM   #6
H_TeXMeX_H
Guru
 
Registered: Oct 2005
Location: $RANDOM
Distribution: slackware64
Posts: 12,928
Blog Entries: 2

Rep: Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269
Quote:
Originally Posted by David1357 View Post
Not very readable.


Even less readable.
Yup, I fully agree, lines like that make me sick.
 
Old 07-13-2009, 05:09 PM   #7
jan61
Member
 
Registered: Jun 2008
Posts: 235

Rep: Reputation: 46
Moin,

*habit*, *not readable*, *makes me sick* - strange. I'd think, that efficiency in scripting should be a part of one's judgement too. Your awk solution is o.k., your sed / cut not, it's a waste of resources. Your *habit* to use redirect makes things more difficult to understand (unnecessary redirects and cat's are like an epidemic in my opinion!).

I only wanted to show, how sed can be used to do the whole work in the second example - if one doesn't understand the line, he should be able to ask. It's a simple basic regular expression, nothing high sophisticated.

Jan
 
Old 07-13-2009, 05:31 PM   #8
David1357
Senior Member
 
Registered: Aug 2007
Location: South Carolina, U.S.A.
Distribution: Ubuntu, Fedora Core, Red Hat, SUSE, Gentoo, DSL, coLinux, uClinux
Posts: 1,302
Blog Entries: 1

Rep: Reputation: 107Reputation: 107
Quote:
Originally Posted by jan61 View Post
I'd think, that efficiency in scripting should be a part of one's judgement too.
The overhead of reading a file from stdin versus opening the file directly is negligible (long.txt is 70007 lines):
Code:
[user@machine:~]:time awk '{ print $3 }' < long.txt >> /dev/null

real    0m0.076s
user    0m0.072s
sys     0m0.004s
[user@machine:~]:time awk '{ print $3 }' long.txt >> /dev/null

real    0m0.077s
user    0m0.060s
sys     0m0.016s
It actually took longer for the version that specified the file name, but I would have to run each command at least 100 times and average the results for us to do a fair comparison.

Quote:
Originally Posted by jan61 View Post
Your awk solution is o.k., your sed / cut not, it's a waste of resources.
Let us see how long the sed-plus-cut solution takes
Code:
[user@machine:~]:time sed -e 's/  */,/g' < long.txt | cut -d ',' -f 3 >> /dev/null

real    0m1.329s
user    0m1.304s
sys     0m0.032s
Much less efficient. So why would I suggest it? If the comma-separated data is saved, then cut can be reused. Let us see how well that works
Code:
[user@machine:~]:time cut -d ',' -f 3 < longc.txt >> /dev/null

real    0m0.042s
user    0m0.040s
sys     0m0.000s
So cut is much more efficient than awk when given well formatted input.

Quote:
Originally Posted by jan61 View Post
Your *habit* to use redirect makes things more difficult to understand (unnecessary redirects and cat's are like an epidemic in my opinion!).
Many would disagree with you. It really comes down to a religious debate.

Quote:
Originally Posted by jan61 View Post
I only wanted to show, how sed can be used to do the whole work in the second example - if one doesn't understand the line, he should be able to ask. It's a simple basic regular expression, nothing high sophisticated.
The problem with those regular expressions you posted is that they are hard to decipher. Someone who writes a script using them may come back in a week and forget what they do. Sometimes script writers like to sacrifice efficiency for clarity.
 
  


Reply

Tags
ksh


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Manipulating Text File with awk or sed kushalkoolwal Programming 2 09-10-2008 08:35 PM
parsing text using sed/awk or similar??? freeindy Programming 5 07-24-2008 05:04 AM
bash/sed/awk fill each line in text file with space to fixed length khairil Programming 11 01-09-2008 06:28 AM
awk/sed to grep the text ahpin Linux - Software 3 10-17-2007 01:34 AM
Help with a script to edit text file (awk? sed?) rickh Linux - Newbie 8 04-21-2005 09:24 PM


All times are GMT -5. The time now is 03:49 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration