LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Blogs > peonuser
User Name
Password

Notices


Rate this Entry

Generic scrapes.

Posted 05-19-2015 at 01:38 PM by peonuser
Updated 05-19-2015 at 02:17 PM by peonuser

Look at astrology as an intellectual cartoon and an insight into human thinking. So I will peek at it once in a while. Also gave a chance to play with page scraping again.

$ ./horoscope.sh Virgo

Daily Horoscope for Tuesday 19th May 2015

Share :

Through friends of someone close, you could learn more about their
background. This extra information, particularly if it's related to how
they acquired their qualifications, and the friendships they made en
route, may not be something you wish to discuss with others, but might go
some way towards explaining why they are pulled towards certain
geographical locations. This might even impact on decisions being taken
now for travel in a couple of months time.

VIRGO

---------------------------------------------


Wrote a script to pull the daily horoscope for a particular sign. The site we are getting the data from has changed. So that led me to go to another site for the time being. Actually it seems a blessing in disguise because now we can get more than the daily listing. Here is the original script.

Original script

Code:
#===================================
# Get today's horoscope
# get sign
hsign=""
read -p "Enter your horscope sign: " hsign
if [ -z $hsign  ]
then hsign="virgo"
fi
# hsign=$(zenity --entry \
#        --title="Daily Horoscope" \
#        --text="Enter your _sign:" \
#        --entry-text "$hsign")
#-------------------------------------------
# output data
# character width required for information box
cw=38
#create data file (datadir and file name can be changed to your needs.
datadir="/home/eddie/bin/signs"
filename="$datadir/th"
# make sure hsign is uppercase
hsign="`echo $hsign|tr '[a-z]' '[A-Z]'`"
cat $datadir/$hsign > $filename
echo -n "Today's date: " >> $filename
date +%D >> $filename
echo "Today's horoscope for:" >> $filename
lynx -width 1000 -dump "http://www.creators.com/lifestylefeatures/horoscopes/horoscopes-by-holiday.html" | grep $hsign | fold -sw $cw >> $filename
#output
# zenity --text-info --filename=$filename
cat $filename
#remove  unneeded file
rm $filename
Was not sure how to do the script, but the I remembered the old CNN script we used. Minor changes and so we are back with the scopes again. To invoke the script you would use ./horoscope.sh signname

$ ./horoscope.sh Virgo

First letter must be capitalized and the rest lower case.

New script

Code:
####################################
# Horoscope  Grabber
#
#===============================
# Assignments
# --------------------------------
datafile="horoscope.txt"
let "flag = 0"
# end assignments
#=================================
#
# Get data file
#---------------------------------
elinks -dump "http://www.horoscopes.co.uk/$1/Daily-Horoscope.php"  > $datafile
#=================================
#
# Extract and display data
#---------------------------------
while read line
do fdata[$a]=$line
    # where to start getting data
    echo $line | grep -q "Daily Horoscope"
    if  [ $? -eq 0 ]; then
        # header
        clear
        let "flag = 1"
    fi
    if [ $flag -eq 1 ]; then
        # where to stop getting data
        echo $line | grep -q "$1"
            if [ $? -eq 0 ]; then
            let "flag = 0"
        else
            echo $line | grep -q "IMG"       
            if [ $? -eq 0 ]; then
                let "response = donothing"
            else
               echo $line | sed 's/\[.*\]//'
           fi
       fi
    fi
let "a += 1"
done < $datafile
# footer
echo ---------------------------------------------
echo
#===================================
# End.
####################################
Posted in Uncategorized
Views 1077 Comments 0
« Prev     Main     Next »
Total Comments 0

Comments

 

  



All times are GMT -5. The time now is 01:26 AM.

Main Menu
Advertisement
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration