Need help writing for loop shell script

hsp40oz · 07-23-2010, 08:20 AM

i am trying to write a script (which I havent done since college) that will grep each line from a file named Backlist which contains the titles of about 850 ebooks against another file named HQNlist which contains about 5500 lines which are book titles plus their corresponding ISBN codes. I want to be able to find those 850 matching titles and write them to a new file because I need to make a list of those ISBNs.

I wrote this script but it just writes the entire HQNlist many times to the new file:

Code:

#!/bin/bash
for line in `cat Backlist`
do
    grep -i $line HQNlist >> results.omfg
done

Any ideas?

Thanks in advance

GrapefruiTgirl · 07-23-2010, 08:26 AM

Code:

cat Backlist | while read line; do
  grep '$line' HQNlist >> results.omfg
done

That should basically do it; if the stuff in the first file (which ends up in the $line variable) contains anything other than "normal" characters like letters & numbers, we may need to enhance the grep a little, but this is the basic idea.

pixellany · 07-23-2010, 08:30 AM

I put your code into [CODE] tags to make it easier to read.

Sasha suggested the same approach that I was going to. To see why the original approach was not working, you could try inserting an echo statement to see the value of "line" each time thru the loop.

hsp40oz · 07-23-2010, 08:45 AM

I tried Sasha's approach. The results were just a blank file. Some of the lines have punctuation.
Is there a way to ignore special characters or should I try to remove all the punctuation from these files?

GrapefruiTgirl · 07-23-2010, 08:47 AM

How about show us a snippet of what's inside each file; that'd be the best way we can ensure that the grepping will work when it encounters odd characters.

schneidz · 07-23-2010, 08:47 AM

whats the content of Backlist ?

could be there are spaces in each line.

maybe you need to do something like: sed -n "$i"p instead.

hsp40oz · 07-23-2010, 09:00 AM

Quote:

Originally Posted by schneidz

whats the content of Backlist ?

could be there are spaces in each line.

maybe you need to do something like: sed -n "$i"p instead.

A few lines from Backlist:
You've Got Male
The Darkest Facts: A Lords of the Underworld Companion
The Darkest Fire
The Amazon's Curse

From HQNlist:
Back on Blossom Street 9781426814686
Last Known Victim 9781426814693
The Healer 9781426814709
Rogue 9781426814716

does that sed command remove the spaces?

ghostdog74 · 07-23-2010, 09:13 AM

use grep's -f option

Code:

grep -f backlist hnlist

ghostdog74 · 07-23-2010, 09:13 AM

*delete*

GrapefruiTgirl · 07-23-2010, 09:15 AM

Code:

echo "$(cat Backlist)" | while read line; do
   grep -e "${line}" HQNlist >> results.omfg
done

Using the data you gave, I've adjusted it and find that the above now works for me. I suspect a problem with newlines and/or spaces, particular spaces, were causing troubles before.

See if that works for you?

SORRY - I initially left out the quotes around the `echo $(..)` which should be there -- please note that and correct..

GrapefruiTgirl · 07-23-2010, 09:18 AM

Quote:

Originally Posted by ghostdog74

use grep's -f option

Code:

grep -f backlist hnlist

ghostdog, I tried this and it didn't work. Any clue why? It'd be great if it did -- nice and simple!

colucix · 07-23-2010, 09:22 AM

Quote:

Originally Posted by GrapefruiTgirl

Using the data you gave, I've adjusted it and find that the above now works for me. I suspect a problem with newlines and/or spaces, particular spaces, were causing troubles before.

Sasha, your previous code (post #2) could work, but you inadvertently embedded $line in single quotes, preventing shell expansion. ghostdog's solution works for me. Cheers!

GrapefruiTgirl · 07-23-2010, 09:25 AM

This is weird colucix

ghost*'s -f idea does not work for me, and neither does my initial suggestion which as you suggest should work with double-quotes (I changed the quotes and still no go).

hsp40oz · 07-23-2010, 09:30 AM

Quote:

Originally Posted by GrapefruiTgirl

Code:

echo "$(cat Backlist)" | while read line; do
   grep -e "${line}" HQNlist >> results.omfg
done

Using the data you gave, I've adjusted it and find that the above now works for me. I suspect a problem with newlines and/or spaces, particular spaces, were causing troubles before.

See if that works for you?

SORRY - I initially left out the quotes around the `echo $(..)` which should be there -- please note that and correct..

this returned 5795 lines which is kind of strange being its a few hundred more than contained in the HQNlist

I wish grep -f worked, it was the first thing i tried last night

colucix · 07-23-2010, 09:31 AM

That's really weird.

Here is my sample session:

Code:

$ Backlist
You've Got Male
The Darkest Facts: A Lords of the Underworld Companion
The Darkest Fire
The Amazon's Curse
$ cat HQNlist
Back on Blossom Street 9781426814686
The Amazon's Curse 9781426816753
Last Known Victim 9781426814693
The Healer 9781426814709
Rogue 9781426814716
The Darkest Facts: A Lords of the Underworld Companion 9781426814728
$ grep -f Backlist HQNlist
The Amazon's Curse 9781426816753
The Darkest Facts: A Lords of the Underworld Companion 9781426814728

and

Code:

$ cat test.sh
#!/bin/bash
cat Backlist | while read line; do
  grep "$line" HQNlist
done
$ ./test.sh
The Darkest Facts: A Lords of the Underworld Companion 9781426814728
The Amazon's Curse 9781426816753