LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 09-24-2008, 09:05 AM   #1
bioinformatics_guy
Member
 
Registered: Aug 2008
Posts: 54

Rep: Reputation: 15
Can I use regexps in linux (sed awk?)


I have a program that prints its output to standard out. What I want to do is look at this data and grab pieces of information from it as it comes.

More specific, if I redirected the output to a file, the last line would be of form:

Final graph has (\d+) nodes and n50 of (\d+) max (\d+)

and I want to grab these 3 numbers and put them in a file. Now, do I have to redirect the output to a file then read through each line or can I just say look at the last line (I'd like to know how to do both for future reference) or can I do it directly from the standard output?
 
Old 09-24-2008, 09:13 AM   #2
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.8, Centos 5.10
Posts: 17,241

Rep: Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325
Either: you can direct to a file and then grep or just pipe through grep eg


yourprog | grep somepattern

or awk or sed; your choice
 
Old 09-24-2008, 09:25 AM   #3
pixellany
LQ Veteran
 
Registered: Nov 2005
Location: Annapolis, MD
Distribution: Arch/XFCE
Posts: 17,802

Rep: Reputation: 738Reputation: 738Reputation: 738Reputation: 738Reputation: 738Reputation: 738Reputation: 738
Regexes are used in a wide variety of BASH utilities (including SED and AWK)

For grabbing numbers out of a line, you can use AWK (specifying which field to print). In your example, it is not clear what the 3 numbers are---or what is the significance of this notation: \d+

Here's a crude example with AWK. (First I created the file "fields"):
Code:
root@ath:/home/mherring/play# more fields
the numbers are 34 456 and 135
root@ath:/home/mherring/play# awk '{print $4, $5, $7}' fields
34 456 135
root@ath:/home/mherring/play#
This of course depends on the numbers always appearing in the same field position.

To print the last line of a file:

sed -n '$p' filename
OR
tail -n1 filename

Go to http://tldp.org and get "Bash Guide for Beginners" and maybe the "Advanced Bash Scripting Guide" (ABS)
 
Old 09-24-2008, 11:05 AM   #4
bioinformatics_guy
Member
 
Registered: Aug 2008
Posts: 54

Original Poster
Rep: Reputation: 15
When I put \d+ I just meant any magnitude of number (0-inf, but it will only be as large as a few thousand.

So, what I'm thinking is redirect the output to a file, tail the file to get the last line then awk it? Does that sound like a reasonable pipeline? If it works Ill place the one liner here.
 
Old 09-24-2008, 11:16 AM   #5
bioinformatics_guy
Member
 
Registered: Aug 2008
Posts: 54

Original Poster
Rep: Reputation: 15
Ok it worked, I printed it out by first redirecting to test, then using the following command

tail -n 1 test | awk '{$4 $9 $11}'

but my output didn't have spaces, say it was 3 45 89, it would print out 34589. How can I add spaces?
 
Old 09-24-2008, 11:29 AM   #6
bioinformatics_guy
Member
 
Registered: Aug 2008
Posts: 54

Original Poster
Rep: Reputation: 15
I also can't redirect the numbers to a file for storing

tail -n 1 test | awk '{$4 $9 $11}' | echo $4 $9 $11 > store

it just prints exit and closes my shell?
 
Old 09-24-2008, 11:31 AM   #7
jschiwal
LQ Guru
 
Registered: Aug 2001
Location: Fargo, ND
Distribution: SuSE AMD64
Posts: 15,733

Rep: Reputation: 670Reputation: 670Reputation: 670Reputation: 670Reputation: 670Reputation: 670
From what I understand of what you want to do, you want to pass the data stream out stdio unchanged but extract the information from it that you need. One way of doing this is to use the "tee" command which is like a tap between two pipes (|). The argument to tee is a file. The standard input is passed on to the standard output and to the file or files which are listed as arguments. The file argument could be a fifo special file, and the fifo file could be used as the file input argument of a sed or awk command running in another script or in background subprocess of the same script.

mkfifo nodes
sed -f extractnodes.sed nodes >extracted_nodes.cvs &
<program generating your data stream> | tee nodes | <program processing your data stream>
 
Old 09-24-2008, 12:14 PM   #8
pixellany
LQ Veteran
 
Registered: Nov 2005
Location: Annapolis, MD
Distribution: Arch/XFCE
Posts: 17,802

Rep: Reputation: 738Reputation: 738Reputation: 738Reputation: 738Reputation: 738Reputation: 738Reputation: 738
Quote:
Originally Posted by bioinformatics_guy View Post
I also can't redirect the numbers to a file for storing

tail -n 1 test | awk '{$4 $9 $11}' | echo $4 $9 $11 > store

it just prints exit and closes my shell?
Within awk, "$4" means the contents of the 4th field.
Outside of awk, "$4" means the contents of the variable named "4".
There's no relationship between the two.

Further, you can't use a pipe to pass the output of awk to echo, since echo does not expect an input.

How about:
tail -n 1 test | awk '{$4 $9 $11}' > store
(you can ALWAYS redirect the output of something to a file.....

(BTW, I did not know awk would work like this without the "print" statement but--as they say--"If it works, it's OK.")
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
sed or awk ilo Programming 1 08-22-2008 11:38 AM
Linux Cheat Sheets (awk, ed, sed, bash, screen, perl) pkrumins Linux - Newbie 1 01-31-2008 04:51 PM


All times are GMT -5. The time now is 01:29 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration