Archive Downloader Using Curl Grep Cygwin

Nimbuz · 04-19-2008, 07:45 PM

Hello,

The following bash script downloads comics from the free 30-day archive. How can the script dowload the comics from membership section using valid username and password? For test purpose only in scripting, the username is user1234 and the password is pass1234. The revised script can login, download, and logout through secure connection. Doable? Thanks in advance.

Code:

#!/bin/bash
# Download comics and put them in the users web directory
# Depends: bsdiff, curl, grep

function download {
  SRC=$1
  FILE=$2
  C_TYPE=`HEAD $SRC|fgrep Content-Type:|cut -d' ' -f2`
  [ ! -e "$FILE" -a "$C_TYPE" != "text/html" ] && \
  wget -q -t inf -A gif,jpg "$SRC" -O "$FILE" && \
  [ x"${FILE##*.}" = x"gif" ] #&& \
#  convert "$FILE" "${FILE%.*}.png" && \
#  rm -f "$FILE"
   echo "$FILE" saved to $PWD ...
}

function changedirectory {

  COMICNAME=$1
  COMICYEAR=$2

  mkdir -p $HOME/public_html/comics/$COMICNAME/$COMICYEAR
  cd $HOME/public_html/comics/$COMICNAME/$COMICYEAR
}

if [ -z "$1" ]
  then
  DATE=`date +"%Y%m%d" -d "-0 day"`
  DATEYR=`date +"%Y" -d "-0 day"`
else
  DATE=$1
fi
mkdir -p $HOME/public_html/comics/
cd $HOME/public_html/comics/

# 9 ChickWeed Lane
changedirectory 9ChickWeedLane $DATEYR
IMG=`curl -s http://www.comics.com/comics/chickweed/archive/chickweed-${DATE}.html \
   | egrep -m 1 -o "/comics/chickweed/archive/images/chickweed[0-9]+.(gif|jpg)"`
download http://www.comics.com$IMG 9ChickWeedLane${DATE}.${IMG##*.}

# Agnes
changedirectory Agnes $DATEYR
IMG=`curl -s http://www.comics.com/creators/agnes/archive/agnes-${DATE}.html \
   | egrep -m 1 -o "/creators/agnes/archive/images/agnes[0-9]+.(gif|jpg)"`
download http://www.comics.com$IMG Agnes${DATE}.${IMG##*.}

# Alley Oop
changedirectory AlleyOop $DATEYR
IMG=`curl -s http://www.comics.com/comics/alleyoop/archive/alleyoop-${DATE}.html \
   | egrep -m 1 -o "/comics/alleyoop/archive/images/alleyoop[0-9]+.(gif|jpg)"`
download http://www.comics.com$IMG AlleyOop${DATE}.${IMG##*.}

MensaWater · 04-19-2008, 07:53 PM

Is the site login done using javascript? If so I don't believe it is doable. Most of the text browsers apparently can't interact with javascript.

Of course it was a couple of years ago when I did my check for curl, wget, lynx, links et all. Maybe someone else will have an answer by now but back then I even ran across one site that said not only didn't such a tool exist but they weren't sure one could ever be made.