SED how to find multiple patterns on a single line

yaazz · 07-28-2009, 09:06 AM

Hi there, I am trying to parse some product titles out of a HTML file that can be obtained at
http://www.moen.com/ecatalog/gallery..._/N-67p?Erp=12

What I am trying to do, is grab the titles of the products (such as fina, mannerly, etc) and print them off one by one. I didnt think this would be difficult at all using regular expressions, but the problem is, that some of the titles are in a huge piece of HTML all stuck on a single line.

Usually I would just use SED, and replace the line with just the part that I wanted, but this wont work in this case since there is a lot of extra crap that will be printed off as well.

My last guess was to use this command
curl 'http://www.moen.com/ecatalog/gallery/bathroom-faucets-sink/_/N-67p?Erp=12' | sed -n 's/.*target="_top"><span class="producttitle">\([a-zA-Z]*\)<\/span>.*/\1/pg'

The output is as follows
Mannerly
Fina
Fina

Which as you can see by looking at the page in a webbrowser, is the last product on each line.
If you take out the .* on each side of the search part of the regex, and look at the output with a fine tooth comb, you can see that this DOES find and replace every regular expression, its just that I dont know how to print them off without all of the extra HTML I dont need.

if this does not make sense please feel free to tell me and I will make myself clearer

paulsm4 · 07-28-2009, 09:47 AM

You might be able to use "sed -e PATTERN1 -e PATTERN2 ...":

http://www.unix.com/shell-programmin...-commands.html

ghostdog74 · 07-28-2009, 07:32 PM

Code:

wget -q -O-   'http://www.moen.com/ecatalog/gallery/bathroom-faucets-sink/_/N-67p?Erp=12' | awk 'BEGIN{RS="</span>"}
/producttitle/{
    gsub(/.*producttitle\">/,"")
    gsub(/<.*/,"")    
    if ($0) print 
}'

yaazz · 07-29-2009, 06:36 AM

Cool thanks a lot ive never seen the gsub operator before in AWK.

What is it doing exactly?
Removing everything before /.*producttitle> then everything after it until the next </span>??

ghostdog74 · 07-29-2009, 09:17 AM

Quote:

Originally Posted by yaazz

Cool thanks a lot ive never seen the gsub operator before in AWK.

most probably, you have been using awk as simple one liners

Quote:

What is it doing exactly?
Removing everything before /.*producttitle> then everything after it until the next </span>??

half right. the second gsub remove everything from "<" (after the text your need to get) till the end.

joeBuffer · 07-29-2009, 09:30 AM

gsub is global substitution, like s/pattern/replacement/g in sed.

yaazz · 07-30-2009, 10:03 AM

Quote:

Originally Posted by ghostdog74

most probably, you have been using awk as simple one liners

half right. the second gsub remove everything from "<" (after the text your need to get) till the end.

Yeah i'm just realizing how powerful AWK is as of late. I always just used sed as it was a lot easier to learn, but after learning some of its amazing features, AWK is soooo much better. I hear perl is very good for text manipulation as well.
What are some other members favorite program?

pixellany · 07-30-2009, 01:56 PM

Quote:

Originally Posted by paulsm4

You might be able to use "sed -e PATTERN1 -e PATTERN2 ...":

http://www.unix.com/shell-programmin...-commands.html

Doesn't this look for PATTERN1 OR PATTERN2?

How about this?:

Code:

sed -n '/P1/{/P2/{/P3/{/P4/p}}}' filename

Keep nesting as required

Quote:

Originally Posted by yaazz

What are some other members favorite program?

Would you believe----SED!!

ghostdog74 · 07-30-2009, 08:30 PM

Quote:

Originally Posted by yaazz

Yeah i'm just realizing how powerful AWK is as of late. I always just used sed as it was a lot easier to learn,

use sed for simple substitution. anything beyond that, use awk (or others)

Quote:

I hear perl is very good for text manipulation as well.

so is Python, or Ruby , or others, with good string manipulation functions/classes. And, its Perl, not perl.

joeBuffer · 07-31-2009, 04:20 AM

I haven't been using them or learning about them for all that long, but I like sed and awk, and anything else that I can put to use.

I've learned about and used a decent amount of Perl, but I couldn't give an opinion except that it's not like it makes me want to avoid learning about sed and awk or anything else.

By the way, Larry Wall has this to say about Perl, on the Linux Journal website:

Quote:

Another interesting tidbit is that the name “perl” wasn't capitalized at first. UNIX was still very much a lower-case-only OS at the time. In fact, I think you could call it an anti-upper-case OS. It's a bit like the folks who start posting on the Net and affect not to capitalize anything. Eventually, most of them come back to the point where they realize occasional capitalization is useful for efficient communication. In Perl's case, we realized about the time of Perl 4 that it was useful to distinguish between “perl” the program and “Perl” the language. If you find a first edition of the Camel Book, you'll see that the title was Programming perl, with a small “p”. Nowadays, the title is Programming Perl.

Also:

Quote:

A couple of years ago, I ran into someone at a trade show who was representing the NSA (National Security Agency). He mentioned to someone else in passing that he'd written a filter program in Perl, so without telling him who I was, I asked him if I could tell people that the NSA uses Perl. His response was, “Doesn't everyone?” So now I don't tell people the NSA uses Perl. I merely tell people the NSA thinks everyone uses Perl. They should know, after all.