LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
Search this Thread
Old 07-23-2010, 08:20 AM   #1
hsp40oz
LQ Newbie
 
Registered: Jul 2010
Posts: 9

Rep: Reputation: 0
Need help writing for loop shell script


i am trying to write a script (which I havent done since college) that will grep each line from a file named Backlist which contains the titles of about 850 ebooks against another file named HQNlist which contains about 5500 lines which are book titles plus their corresponding ISBN codes. I want to be able to find those 850 matching titles and write them to a new file because I need to make a list of those ISBNs.

I wrote this script but it just writes the entire HQNlist many times to the new file:

Code:
#!/bin/bash
for line in `cat Backlist`
do
    grep -i $line HQNlist >> results.omfg
done
Any ideas?

Thanks in advance

Last edited by pixellany; 07-23-2010 at 08:27 AM.
 
Old 07-23-2010, 08:26 AM   #2
GrapefruiTgirl
Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550
Code:
cat Backlist | while read line; do
  grep '$line' HQNlist >> results.omfg
done
That should basically do it; if the stuff in the first file (which ends up in the $line variable) contains anything other than "normal" characters like letters & numbers, we may need to enhance the grep a little, but this is the basic idea.
 
Old 07-23-2010, 08:30 AM   #3
pixellany
LQ Veteran
 
Registered: Nov 2005
Location: Annapolis, MD
Distribution: Arch/XFCE
Posts: 17,802

Rep: Reputation: 728Reputation: 728Reputation: 728Reputation: 728Reputation: 728Reputation: 728Reputation: 728
I put your code into [CODE] tags to make it easier to read.

Sasha suggested the same approach that I was going to. To see why the original approach was not working, you could try inserting an echo statement to see the value of "line" each time thru the loop.
 
Old 07-23-2010, 08:45 AM   #4
hsp40oz
LQ Newbie
 
Registered: Jul 2010
Posts: 9

Original Poster
Rep: Reputation: 0
I tried Sasha's approach. The results were just a blank file. Some of the lines have punctuation.
Is there a way to ignore special characters or should I try to remove all the punctuation from these files?
 
Old 07-23-2010, 08:47 AM   #5
GrapefruiTgirl
Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550
How about show us a snippet of what's inside each file; that'd be the best way we can ensure that the grepping will work when it encounters odd characters.
 
Old 07-23-2010, 08:47 AM   #6
schneidz
Senior Member
 
Registered: May 2005
Location: boston, usa
Distribution: fc-15/ fc-20-live-usb/ aix
Posts: 4,023

Rep: Reputation: 625Reputation: 625Reputation: 625Reputation: 625Reputation: 625Reputation: 625
whats the content of Backlist ?

could be there are spaces in each line.

maybe you need to do something like: sed -n "$i"p instead.
 
Old 07-23-2010, 09:00 AM   #7
hsp40oz
LQ Newbie
 
Registered: Jul 2010
Posts: 9

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by schneidz View Post
whats the content of Backlist ?

could be there are spaces in each line.

maybe you need to do something like: sed -n "$i"p instead.
A few lines from Backlist:
You've Got Male
The Darkest Facts: A Lords of the Underworld Companion
The Darkest Fire
The Amazon's Curse

From HQNlist:
Back on Blossom Street 9781426814686
Last Known Victim 9781426814693
The Healer 9781426814709
Rogue 9781426814716

does that sed command remove the spaces?
 
Old 07-23-2010, 09:13 AM   #8
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,695
Blog Entries: 5

Rep: Reputation: 241Reputation: 241Reputation: 241
use grep's -f option
Code:
grep -f backlist hnlist
 
Old 07-23-2010, 09:13 AM   #9
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,695
Blog Entries: 5

Rep: Reputation: 241Reputation: 241Reputation: 241
*delete*

Last edited by ghostdog74; 07-23-2010 at 09:15 AM.
 
Old 07-23-2010, 09:15 AM   #10
GrapefruiTgirl
Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550
Code:
echo "$(cat Backlist)" | while read line; do
   grep -e "${line}" HQNlist >> results.omfg
done
Using the data you gave, I've adjusted it and find that the above now works for me. I suspect a problem with newlines and/or spaces, particular spaces, were causing troubles before.

See if that works for you?

SORRY - I initially left out the quotes around the `echo $(..)` which should be there -- please note that and correct..

Last edited by GrapefruiTgirl; 07-23-2010 at 09:21 AM.
 
Old 07-23-2010, 09:18 AM   #11
GrapefruiTgirl
Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550
Quote:
Originally Posted by ghostdog74 View Post
use grep's -f option
Code:
grep -f backlist hnlist
ghostdog, I tried this and it didn't work. Any clue why? It'd be great if it did -- nice and simple!
 
Old 07-23-2010, 09:22 AM   #12
colucix
Moderator
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,503

Rep: Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957
Quote:
Originally Posted by GrapefruiTgirl View Post
Using the data you gave, I've adjusted it and find that the above now works for me. I suspect a problem with newlines and/or spaces, particular spaces, were causing troubles before.
Sasha, your previous code (post #2) could work, but you inadvertently embedded $line in single quotes, preventing shell expansion. ghostdog's solution works for me. Cheers!
 
Old 07-23-2010, 09:25 AM   #13
GrapefruiTgirl
Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550Reputation: 550
This is weird colucix

ghost*'s -f idea does not work for me, and neither does my initial suggestion which as you suggest should work with double-quotes (I changed the quotes and still no go).
 
Old 07-23-2010, 09:30 AM   #14
hsp40oz
LQ Newbie
 
Registered: Jul 2010
Posts: 9

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by GrapefruiTgirl View Post
Code:
echo "$(cat Backlist)" | while read line; do
   grep -e "${line}" HQNlist >> results.omfg
done
Using the data you gave, I've adjusted it and find that the above now works for me. I suspect a problem with newlines and/or spaces, particular spaces, were causing troubles before.

See if that works for you?

SORRY - I initially left out the quotes around the `echo $(..)` which should be there -- please note that and correct..
this returned 5795 lines which is kind of strange being its a few hundred more than contained in the HQNlist

I wish grep -f worked, it was the first thing i tried last night
 
Old 07-23-2010, 09:31 AM   #15
colucix
Moderator
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,503

Rep: Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957
That's really weird.

Here is my sample session:
Code:
$ Backlist
You've Got Male
The Darkest Facts: A Lords of the Underworld Companion
The Darkest Fire
The Amazon's Curse
$ cat HQNlist
Back on Blossom Street 9781426814686
The Amazon's Curse 9781426816753
Last Known Victim 9781426814693
The Healer 9781426814709
Rogue 9781426814716
The Darkest Facts: A Lords of the Underworld Companion 9781426814728
$ grep -f Backlist HQNlist
The Amazon's Curse 9781426816753
The Darkest Facts: A Lords of the Underworld Companion 9781426814728
and
Code:
$ cat test.sh
#!/bin/bash
cat Backlist | while read line; do
  grep "$line" HQNlist
done
$ ./test.sh
The Darkest Facts: A Lords of the Underworld Companion 9781426814728
The Amazon's Curse 9781426816753
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
shell script question $variable in loop icecubeflower Linux - Newbie 2 03-31-2009 09:09 AM
shell script , while loop ykc Programming 5 03-30-2009 07:50 AM
Shell Script skipping a loop dnvikram Programming 2 01-23-2009 02:29 PM
Loop in Shell Script delamatrix Programming 4 07-24-2008 05:20 PM
optional exit from loop, shell script RudraB Programming 2 07-17-2008 03:30 AM


All times are GMT -5. The time now is 01:42 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration