LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 11-29-2011, 10:57 AM   #1
johannni
LQ Newbie
 
Registered: Nov 2011
Posts: 5

Rep: Reputation: Disabled
indexing list of words in a file


Hey all,

I'm doing a project currently and want to index words in a webpage.
So there would be a file with webpage content and a file with list of words, I want an output file with true and false that would show which word exists in the webpage.

example:

Webpage content data.html




Code:
References

   1. http://console.online.net/
   2. http://webmail.online.net/
   3. http://console.online.net/assistance/
   4. http://www.online.net/
   5. http://www.online.net/nom-de-domaine/comparatif-des-extensions-geographiques.xhtml
   6. http://www.online.net/nom-de-domaine/comparatif-des-extensions-geographiques.xhtml
   7. http://console.online.net/commande/index/
   8. http://www.online.net/hebergement-mutualise/comparatif-des-offres-pour-site-internet.xhtml
   9. http://www.online.net/hebergement-mutualise/comparatif-des-offres-pour-site-internet.xhtml
  10. http://www.online.net/hebergement-mutualise/offre-online-basic.xhtml
  11. http://www.online.net/hebergement-mutualise/offre-online-pro.xhtml
  12. http://www.online.net/hebergement-mutualise/offre-online-illimite.xhtml
  13. http://www.online.net/serveur-dedie/comparatif-offres-serveur-dedie.xhtml
  14. http://www.online.net/serveur-dedie/comparatif-serveur-dedie-start.xhtml
  15. http://www.online.net/serveur-dedie/offre-dedibox-sc.xhtml
  16. http://www.online.net/serveur-dedie/offre-dedibox-classic.xhtml
  17. http://www.online.net/serveur-dedie/offre-dedibox-dc.xhtml
  18. http://www.online.net/serveur-dedie/offre-dedibox-qc.xhtml
  19. http://www.online.net/serveur-dedie/comparatif-serveur-dedie-pro.xhtml
  20. http://www.online.net/serveur-dedie/offre-dedibox-pro-r210.xhtml
  21. http://www.online.net/serveur-dedie/offre-dedibox-pro-r410.xhtml
  22. http://www.online.net/serveur-dedie/offre-dedibox-pro-r510.xhtml
  23. http://www.online.net/serveur-dedie/offre-dedibox-storage.xhtml
  24. http://www.online.net/serveur-dedie/offre-dedibox-housing-dedirack.xhtml
  25. http://www.online.net/serveur-dedie/offre-dedibox-housing-dedirack.xhtml
  26. http://www.iliad-entreprises.fr/
  27. http://www.online.net/infogerance-serveur/infogerance-serveur-dedie.xhtml
  28. http://www.iliad-datacenter.fr/
  29. https://console.online.net/commande/server/?server=110
  30. http://www.online.net/
  31. http://console.online.net/assistance/
  32. http://twitter.com/online_fr
  33. http://www.online.net/hebergement-mutualise/comparatif-des-offres-pour-site-internet.xhtml
  34. https://console.online.net/commande/index/
  35. http://www.online.net/serveur-dedie/comparatif-serveur-dedie-start.xhtml
  36. https://console.online.net/commande/server/?server=110
  37. http://www.online.net/fiche-tarifaire.pdf
  38. http://www.online.net/cgv.pdf
  39. http://www.online.net/document-legal/mentions-legales.xhtml
  40. http://www.online.net/
list of words words.dat



Code:
online
hebergement
ftp
35
php
.fr
.se

file with true false that would show the existence of the words
output.dat


Code:
true
false
true
false
true
false

thnx
 
Old 11-29-2011, 11:29 AM   #2
thesnow
Member
 
Registered: Nov 2010
Location: Minneapolis, MN
Distribution: Ubuntu, Red Hat, Mint
Posts: 170

Rep: Reputation: 56
Code:
#!/bin/bash

for i in `cat words.dat`
do
	if grep -q "$i" data.html
	then
		echo "true"
	else
		echo "false"
	fi
done
redirect to output.dat
 
Old 11-29-2011, 12:05 PM   #3
johannni
LQ Newbie
 
Registered: Nov 2011
Posts: 5

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by thesnow View Post
Code:
#!/bin/bash

for i in `cat words.dat`
do
	if grep -q "$i" data.html
	then
		echo "true"
	else
		echo "false"
	fi
done
redirect to output.dat

It's working, but I'm recieving more lines of output that the lines in the words.dat, do you think the grep looks for all numbers of matching and for example if there was two matching each pattern it would return two "true" ?
 
Old 11-29-2011, 01:03 PM   #4
thesnow
Member
 
Registered: Nov 2010
Location: Minneapolis, MN
Distribution: Ubuntu, Red Hat, Mint
Posts: 170

Rep: Reputation: 56
I get 7 lines of output--how many do you get?

Code:
[user01@lm ~/testing] $ wc -l words.dat 
7 words.dat
[user01@lm ~/testing] $ ./found.bash | wc -l
7
 
Old 11-29-2011, 02:37 PM   #5
johannni
LQ Newbie
 
Registered: Nov 2011
Posts: 5

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by thesnow View Post
I get 7 lines of output--how many do you get?

Code:
[user01@lm ~/testing] $ wc -l words.dat 
7 words.dat
[user01@lm ~/testing] $ ./found.bash | wc -l
7
My input is big:

Code:
General  
Buttons  
Back  Camera  Home  Menu  Search  Volume  
Developed especially for seniors  
No  
Java  
Yes  
Java MIDP  
null  
Operating system  
Android 2.3 (Gingerbread)  
Qwertykeyboardbuttons  
No  
Qwertykeyboardscreen  
Yes  
SAR  
0.93 W/kg  
Stand by time  
700 h  
Talk time  
8 h  
Talktime3G  
7.5 h  
StandByTime3G  
550 h  
Audio features  
Built-in speakers  
Mono  
Built-in camera  
Yes  
Auto focus  
Yes  
Built-in flash  
No  
Digital zoomstill image  
null  
Effective pixels  
3 megapixels  
Maximumresolutionstill  
2048x1536 pixels  
Number of cameras  
1 pc  
Optical zoom  
No  
Video recorder  
Yes  
Digital zoomvideo  
null  
Maximum FPS when recording  
18 fps  
Maximumresolutionvideo  
640x480 pixels  
Lamp  
No  
Communication  
3GUMTS  
Yes  
HSPA  
Yes  
Speeddown  
7.2 Mbit/s  
Speedup  
2 Mbit/s  
Bluetooth  
Yes  
Bluetooth profile  
A2DP  
Version  
2.1+EDR  
Built-in FM transmitter  
No  
EDGE  
Yes  
Frequency band  
Quad  
GPRS  
Yes  
IR  
No  
MMS  
Yes  
Push-to-talk  
No  
Support for IP telephony  
Yes  
Built-in SIP stack  
Yes  
Support via software  
Yes  
Support for Microsoft Exchange  
Yes  
UMA  
No  
WAP  
Yes  
Version  
2.0  
Web and email functions  
Browser  Email Client  Push mail  
Wireless network  
Yes  
Supports media sharing  
No  
Type  
802.11b  802.11g  
Compatibility  
DLNA certified  
No  
OpenGL ES  
Yes  
VersionOpenGLES  
2.0  
Connectors  
3.5mm headphone output  
Yes  
Type of connection  
3.5 mm  
USB  
Yes  
Charging via USB  
Yes  
Type of USB connector  
Micro-USB  
USB version  
2.0  
Video output  
No  
Decoders  
FM radio  
Yes  
RDS radio  
Yes  
Visual Radio  
No  
Supported audio formats  
AAC  AAC+  eAAC+  MP3  WAV (PCM)  WMA  
Supported image files  
JPEG / JPG  
Supported video files  
DivX  H.263  H.264  MPEG4  WMV  
Design  
Colour  
Black  Gray  Silver  White  
Foldable  
No  
Sliding  
No  
Wristwatch  
No  
Dimensions  
DimensionsDimensions  
59  113.5  13.3 mm  
Weight  
129 g  
Features  
Built in Positioning navigation  
Yes  
Integrated geotagging  
Yes  
type of positioninng  
A-GPS  GPS  
Built-in internet services  
Facebook  Flickr  Picasa  Skype  Twitter  YouTube  
Features  
Alarm Clock  Calculators  Calendar  Dictaphone  Games  Media player  
T9  Vibrator  Watch  
Flightmode  
Yes  
Shock resistant  
No  
Speaker function  
Yes  
Support for dual SIM cards  
No  
Voice dialing  
null  
Image properties  
Display  
Colour  
Number of colours  
262144 colours  
Display technology  
LCD  
Dual screens  
No  
Pivot function  
Yes  
Type  
Automatic  
Scratch resistant display  
No  
Screen resolution  
320x480 pixels  
Screen size  
3.2 inches  
Touchscreen  
Yes  
Simultaneous touch points  
3+ Multi-touch  
Type of technology  
Capacitive  
Materials  
Materialshell  
null  
Power supply  
Battery capacity  
1500 mAh  
Processor  
Graphics processor  
Yes  
Graphics processor type  
Adreno 200 (AMD z430)  
Processor core type  
Single Core  
Processor speed  
600 Mhz  
Processor type  
ARM11  
RAM  
Yes  
Memory quantity  
512 MB  
SoCSystemonChip  
Qualcomm MSM 7227  
Storage  
Internal storage space  
170 MB  
Memory card reader  
Yes  
SDIO  
null  
Compatible memory card types  
microSD/TransFlash  microSDHC  
Including memory card  
Yes  
Including memory quantity  
2GB  
Maximum amount of memory  
32GB  
ROM memory  
Yes  
Memory quantity  
170 MB  
Miscellaneous  
Release year  
2010


My word list:

Code:
802.11a  
802.11b  
802.11g  
802.11n  
Player  
Server  
Controller  
Uploader  
Dowloader  
MP3  
WMA  
OGG Vorbis  
FLAC  
WAV (PCM)  
AAC  
AMR  
AIFF  
AAC+  
eAAC+  
Matroska (MKA)  
Apple Lossless  
JPEG / JPG  
TIFF  
PNG  
BMP  
GIF  
DivX  
Xvid  
MPEG1  
MPEG2  
MPEG4  
WMV  
H.263  
3GP/3GPP  
H.264  
WMV9 HD (VC-1)  
AVI  
Matroska (MKV)  
AVC(HD)  
MOV  
TS  
DivX HD  
Real Media (RM / RMVB)  
FLV  
Black  
Silver  
Blue  
White  
Red  
Green  
Gray  
Yellow  
Brown  
Purple  
Orange  
Beige  
Gold  
Pink  
Bronze  
Copper  
A-GPS  
GPS  
Facebook  
YouTube  
Twitter  
Skype  
Flickr  
Picasa  
Myspace  
Games  
Calendar  
Media player  
Dictaphone  
Calculators  
Stopwatch  
Notebook  
Vibrator  
Watch  
Currency Converter  
T9  
Digital TV (DVB-H)  
Alarm Clock  
Timer  
TrackID  
Compass  
Supports NFC  
ANT+  
Key pad  
Camera  
Play / Pause  
Next song  
Previous song  
Volume  
Music player  
Keyguard  
Home  
Search  
Back  
Menu  
xD-Picture  
microSD/TransFlash  
Multi Media Card (MMC)  
Secure Digital (SD)  
miniSD  
Memory Stick Pro Duo (MS Pro Duo)  
Memory Stick Duo (MS Duo)  
Memory Stick Micro (MS Micro)  
RS-Multi Media Card (RS-MMC)  
microSDHC  
Email Client  
Browser  
Push mail

and my output:

Code:
false
true
true
false
false
false
false
false
false
true
true
false
false
false
true
true
true
false
false
true
true
false
false
false
false
true
true
true
false
false
false
false
true
false
false
false
true
true
true
false
true
false
false
false
false
false
false
false
false
true
true
false
false
true
false
true
false
false
true
true
true
true
false
false
true
false
false
false
false
false
false
false
false
false
true
true
true
true
true
true
true
true
false
true
true
true
true
true
true
false
false
true
true
false
false
true
true
false
false
true
true
false
false
false
true
false
false
false
false
true
false
true
false
false
false
false
false
true
false
true
false
true
true
true
true
false
true
true
true
false
false
false
true
false
false
true
false
true
false
false
true
false
true
false
false
false
false
true
false
true
false
false
false
true
false
false
true
true
true
true
true
true
which the words has 120 lines and the output has 163 lines, I tried to fix it but didn't worked

Do you have any idea about that?
 
Old 11-29-2011, 06:47 PM   #6
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,256

Rep: Reputation: 2686Reputation: 2686Reputation: 2686Reputation: 2686Reputation: 2686Reputation: 2686Reputation: 2686Reputation: 2686Reputation: 2686Reputation: 2686Reputation: 2686
The script works with the original example because you did not have any lines with multiple words on a line. A for loop will split each line based on the IFS
builtin variable and hence any line that has a space in it will produce multiple results.

As this is a project / homework I will simply suggest you look into the while loop construct for reading the values from the file.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
looking for a program to display random words from list newbiesforever Linux - Software 10 04-23-2011 02:21 PM
Python: replacing all words found on a list General Programming 3 03-15-2010 09:57 AM
recoll indexing error while indexing Mail/ [SOLVED] ajnabi Linux - Software 1 09-14-2009 01:44 AM
Getting Range of Words from Arg List tonyfreeman Programming 3 10-05-2006 04:35 PM
Sorting a list of words in LISP! Hady Programming 1 05-01-2004 04:29 PM


All times are GMT -5. The time now is 11:05 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration